Skip to contentSkip to conversation

mm/swap: priority-based swap tiers with per-cgroup selection

RFC PATCH In review Youngjun Park 4 participants 11 messages Sep 16 – Sep 26

Subsystems (5)
lore
Share & tools
lore

Youngjun Park

author reviewer
#0 lore raw

v11 cover mm/swap: priority-based swap tiers with per-cgroup selection

This is RFC v11 of the swap tier series [1], reworked after the v10 discussion [2][3].

Motivation
==========

After the v10 discussion, I would like to first land the parts that do not affect users, before anything touches memcg, with two goals in mind.

- Serve as a bridge that causes no trouble when the tier interface and memcg support are introduced later.
- Make per-cgroup swap device selection possible.

While thinking this through, I came across some insights that I'd like to discuss with the community. Hence this RFC v11.

As a starting point, let's look at what swap priority already gives us.

- Devices with different priorities are used in priority order.
- Devices with the same priority are used round-robin.

From this point of view, a priority itself can be seen as a tier rather than a property of a device. Each distinct priority is a tier, the devices with that priority hang off it, and allocation walks the tiers.

This is one of the insights that led me here. In the v10 design, a tier was a priority range, so one tier could contain multiple priorities. That raised two questions.

- If devices in a tier are allocated in priority order, aren't they effectively different tiers?
- Is there a real use case for several same-priority round-robin groups inside one tier?

If each distinct priority is its own tier, both questions go away. A tier is one round-robin group, and ordering exists only between tiers. (Anyone who wants priority-based allocation inside a tier can still get it through a separate tier interface later.
mix allocation policy? also possible. we have interface.)

Either way, no explicit tier interface like the one in v10 is needed yet.

Even when real tiers work are introduced, the view stays the same.

- We want an ordering between tier A and tier B.
- Devices in one tier form one service speed group. How they share allocations may become flexible, and will normally stay round-robin as it is today.
- On top of that, we will add backend transfer between tiers after swap virtualization.

This series changes the current swap code to follow that view, which gives us the following.

- Per-cgroup swap can later fit into a swap tier interface without trouble.
- A base for the per-priority allocation queue series [4].
- It is close to a refactoring of the existing structure, so userspace is not tied to an implementation.

Kairui, Lian, would it be okay to use this as the groundwork for [4]? What do you think?

Per-cgroup swap in debugfs
==========================

Patches 3 and 4 let a memory cgroup choose its tiers through debugfs.

  # swapon -p 100 /dev/nvme0n1p2
  # swapon -p 50 /dev/sdb2
  # cat /sys/kernel/debug/swap/tiers
  Idx   Prio
  0     100
  1     50
  # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers

Bit i of the mask is tier i, so /batch swaps only to sdb2. A tier keeps its index for its lifetime, so the mask keeps selecting the same tier across swapon and swapoff.

Masks are kept in a list keyed by cgroup ID rather than in struct mem_cgroup, so the series does not touch memcg code and needs no Kconfig option. A mask applies only to its own cgroup and is not inherited by child cgroups.

This debugfs interface is a stepping stone. Once the tier model settles, /sys/kernel/mm/swap/tiers would list and name the tiers, and a memcg knob such as memory.swap.tiers.max would take tier names and turn them into this mask.

Future direction
================

This is not done in this series. If the tiers introduced here settle, which differs from giving tiers priority ranges, work can continue along these lines depending on the discussion.

The following are common to both.
- The per-priority allocation queue series from Kairui and Lian [4].
- Add /sys/kernel/mm/swap/tiers.
- A memcg memory.swap.tiers.max interface.
- Per-tier allocation policy (an extension that needs discussion).

If tiers are assigned by priority ranges as before, the tiers here have to be related to those ranges. When a tier is assigned at runtime, the devices that fall into its range are merged into that tier.

If tiers are extended the way they work here, a tier stays a priority and the interface only adds names. A tier starts with a default name, such as its priority, and can be renamed through the tier interface.

(I think there are a lot of possible way which we can discuss.)

Change log
==========

v11
- Reworked after the v10 discussion [2][3], with no new user ABI.
- A swap tier is now a single swap priority, created and removed by
swapon and swapoff. /sys/kernel/mm/swap/tiers and CONFIG_NR_SWAP_TIERS
are dropped.
- The active and available swap device lists move into the tiers.
(#1, #2 patches)
- Per-cgroup tier selection moves from memory.swap.tiers.max to debugfs.
(#3 patch)
- Masks are kept in a cgroup ID list instead of struct mem_cgroup, and
are no longer inherited by child cgroups. (#3 patch)
- Dropped the selftests, which tested the dropped interfaces.
- Rebased on recent mm-new.
- v10 link: https://lore.kernel.org/linux-mm/[email protected]/

Changes up to v10 are in the v10 cover letter [1].

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/

Youngjun Park (4):
mm: swap: introduce swap tier infrastructure
mm: swap: allocate swap slots from swap tiers
mm: swap: add a debugfs interface for memcg tier selection
mm: swap: filter swap allocation by memcg tier mask

 Documentation/mm/index.rst     |   1 +
 Documentation/mm/swap-tier.rst |  55 +++++
 MAINTAINERS                    |   3 +
 include/linux/swap.h           |   5 +-
 mm/Makefile                    |   2 +-
 mm/swap.h                      |   2 +
 mm/swap_tier.c                 | 421 +++++++++++++++++++++++++++++++++
 mm/swap_tier.h                 |  71 ++++++
 mm/swapfile.c                  | 173 ++++++++------
 9 files changed, 658 insertions(+), 75 deletions(-)
 create mode 100644 Documentation/mm/swap-tier.rst
 create mode 100644 mm/swap_tier.c
 create mode 100644 mm/swap_tier.h

base-commit: b08a65b93426d86e3f354d655d6225397b591877

Signature · 1 line
2.48.1
author reviewer
#1 lore raw

RFC PATCH v11 1/4 mm: swap: introduce swap tier infrastructure

Introduce the "swap tier" as an abstraction over swap devices, so that swap allocation is organized on a tier basis rather than on a flat list of devices.

Swap priority already points this way. Devices with different priorities are used in priority order, and devices with the same priority are used round-robin. A priority therefore already behaves like a tier, a group of devices that share one service speed. This patch gives that group an explicit structure, without adding any user interface.

A swap tier is the set of swap devices that share a priority. A tier is created when the first device with its priority is swapped on, and removed when the last one is swapped off. The active tiers are kept sorted by priority for allocation.

Making the tier own its devices gives a same-priority group the data structures it needs and lays the groundwork for tier-based allocation, onto which per-cgroup swap device selection can later be fit.

No tier feature and no user-visible change are introduced here. This only prepares the ground for them.

---
 MAINTAINERS          |   2 +
 include/linux/swap.h |   2 +-
 mm/Makefile          |   2 +-
 mm/swap.h            |   1 +
 mm/swap_tier.c       | 114 +++++++++++++++++++++++++++++++++++++++++++
 mm/swap_tier.h       |  37 ++++++++++++++
 mm/swapfile.c        |  66 +++++++++++++------------
 7 files changed, 192 insertions(+), 32 deletions(-)
 create mode 100644 mm/swap_tier.c
 create mode 100644 mm/swap_tier.h
MAINTAINERS +2 -0
index e4412c3d8d45..37f353015cae 100644
@@ -17410,6 +17410,8 @@ F:	mm/page_io.c
 F:	mm/swap.h
 F:	mm/swap_table.h
 F:	mm/swap_state.c
+F:	mm/swap_tier.c
+F:	mm/swap_tier.h
 F:	mm/swapfile.c
 
 MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
include/linux/swap.h +1 -1
index 43155e122b5c..22ccb4b5801e 100644
@@ -237,7 +237,7 @@ struct swap_info_struct {
 	struct percpu_ref users;	/* indicate and keep swap device valid. */
 	unsigned long	flags;		/* SWP_USED etc: see above */
 	signed short	prio;		/* swap priority of this type */
-	struct plist_node list;		/* entry in swap_active_head */
+	struct plist_node list;		/* entry in its swap tier */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
 	struct swap_cluster_info *cluster_info; /* array, one entry per cluster */
mm/Makefile +1 -1
index 2a3ec53d62ee..d89a7abadc46 100644
@@ -76,7 +76,7 @@ ifdef CONFIG_MMU
 	obj-$(CONFIG_ADVISE_SYSCALLS)	+= madvise.o
 endif
 
-obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o
+obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o swap_tier.o
 obj-$(CONFIG_ZSWAP)	+= zswap.o
 obj-$(CONFIG_HAS_DMA)	+= dmapool.o
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o
mm/swap.h +1 -0
index b3b54c28929a..4de6b9b0f261 100644
@@ -36,6 +36,7 @@ struct swap_io_ctx;
 #define swap_entry_order(order)	0
 #endif
 
+extern spinlock_t swap_lock;
 extern struct swap_info_struct *swap_info[];
 
 /*
mm/swap_tier.c new file +114 -0

114 lines not on this page · raw message

mm/swap_tier.h new file +37 -0
new file mode 100644
index 000000000000..3dce716d23f6
@@ -0,0 +1,37 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _SWAP_TIER_H
+#define _SWAP_TIER_H
+
+#include <linux/list.h>
+#include <linux/plist.h>
+#include <linux/types.h>
+
+/* Forward declarations */
+struct swap_info_struct;
+
+/*
+ * struct swap_tier - structure representing a swap tier.
+ *
+ * @prio: priority of the swap devices in the tier.
+ * @active_head: swap devices in the tier.
+ * @list: linkage into swap_tier_active_list or swap_tier_inactive_list.
+ */
+struct swap_tier {
+	short prio;
+	struct plist_head active_head;
+	struct list_head list;
+};
+
+extern struct list_head swap_tier_active_list;
+
+#define for_each_active_tier(tier) \
+	list_for_each_entry(tier, &swap_tier_active_list, list)
+
+/* Initialization and application */
+void swap_tiers_init(void);
+
+/* Tier assignment */
+void swap_tiers_assign_dev(struct swap_info_struct *swp);
+void swap_tiers_remove_dev(struct swap_info_struct *swp);
+
+#endif /* _SWAP_TIER_H */
mm/swapfile.c +36 -30

136 lines not on this page · raw message

Signature · 1 line
2.48.1
author reviewer
#2 lore raw

RFC PATCH v11 2/4 mm: swap: allocate swap slots from swap tiers

In reply to Youngjun Park

The slow allocation path walks swap_avail_head, one plist of every available swap device, ordered by priority.

Give each tier its own list of available devices instead. The slow path now walks the tiers in order, and walks each tier's plist as before. __folio_throttle_swaprate() walks them the same way.

swap_avail_lock still protects these lists. The tier list changes under it too, so the walk can go on after it drops the lock, as long as its tier is still there.

This is the groundwork for the per-priority allocation queue series from Kairui and Lian [1]. That series can move onto the per-tier device lists one tier at a time.

[1] https://lore.kernel.org/linux-mm/[email protected]/

Assisted-by Claude:claude-opus-5
---
 include/linux/swap.h |  2 +-
 mm/swap.h            |  1 +
 mm/swap_tier.c       | 17 +++++++++-
 mm/swap_tier.h       |  3 ++
 mm/swapfile.c        | 78 ++++++++++++++++++++++++--------------------
 5 files changed, 64 insertions(+), 37 deletions(-)
include/linux/swap.h +1 -1
index 22ccb4b5801e..df69c2dd434a 100644
@@ -268,7 +268,7 @@ struct swap_info_struct {
 	struct work_struct discard_work; /* discard worker */
 	struct work_struct reclaim_work; /* reclaim worker */
 	struct list_head discard_clusters; /* discard clusters list */
-	struct plist_node avail_list;   /* entry in swap_avail_head */
+	struct plist_node avail_list;   /* entry in its tier's avail list */
 	const struct swap_ops *ops;
 };
 
mm/swap.h +1 -0
index 4de6b9b0f261..916c0ad128f7 100644
@@ -37,6 +37,7 @@ struct swap_io_ctx;
 #endif
 
 extern spinlock_t swap_lock;
+extern spinlock_t swap_avail_lock;
 extern struct swap_info_struct *swap_info[];
 
 /*
mm/swap_tier.c +16 -1
index 8ed1427cee09..286f319fb125 100644
@@ -65,6 +65,7 @@ void swap_tiers_init(void)
 
 	for_each_tier(tier, idx) {
 		plist_head_init(&tier->active_head);
+		plist_head_init(&tier->avail_head);
 		INIT_LIST_HEAD(&tier->list);
 		swap_tier_inactivate(tier);
 	}
@@ -92,11 +93,14 @@ void swap_tiers_assign_dev(struct swap_info_struct *swp)
 
 	lockdep_assert_held(&swap_lock);
 
+	/* The allocator walks the tiers under swap_avail_lock. */
+	spin_lock(&swap_avail_lock);
 	tier = swap_tier_lookup(swp->prio);
 	if (!tier) {
 		tier = swap_tier_prepare(swp->prio);
 		swap_tier_activate(tier);
 	}
+	spin_unlock(&swap_avail_lock);
 
 	plist_add(&swp->list, &tier->active_head);
 }
@@ -109,6 +113,17 @@ void swap_tiers_remove_dev(struct swap_info_struct *swp)
 
 	tier = swap_tier_lookup(swp->prio);
 	plist_del(&swp->list, &tier->active_head);
-	if (plist_head_empty(&tier->active_head))
+	if (plist_head_empty(&tier->active_head)) {
+		spin_lock(&swap_avail_lock);
 		swap_tier_inactivate(tier);
+		spin_unlock(&swap_avail_lock);
+	}
+}
+
+/* The avail list of the tier @swp belongs to. */
+struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp)
+{
+	lockdep_assert_held(&swap_avail_lock);
+
+	return &swap_tier_lookup(swp->prio)->avail_head;
 }
mm/swap_tier.h +3 -0
index 3dce716d23f6..c4347c28d8f1 100644
@@ -14,11 +14,13 @@ struct swap_info_struct;
  *
  * @prio: priority of the swap devices in the tier.
  * @active_head: swap devices in the tier.
+ * @avail_head: available swap devices in the tier.
  * @list: linkage into swap_tier_active_list or swap_tier_inactive_list.
  */
 struct swap_tier {
 	short prio;
 	struct plist_head active_head;
+	struct plist_head avail_head;
 	struct list_head list;
 };
 
@@ -33,5 +35,6 @@ void swap_tiers_init(void);
 /* Tier assignment */
 void swap_tiers_assign_dev(struct swap_info_struct *swp);
 void swap_tiers_remove_dev(struct swap_info_struct *swp);
+struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp);
 
 #endif /* _SWAP_TIER_H */
mm/swapfile.c +43 -35

126 lines not on this page · raw message

Signature · 1 line
2.48.1
author reviewer
#3 lore raw

RFC PATCH v11 3/4 mm: swap: add a debugfs interface for memcg tier selection

In reply to Youngjun Park

Swap tiers do nothing until something chooses between them. Let a memory cgroup do that.

Each cgroup carries a mask of the tiers it may swap to. Write a cgroup path and a hex mask to /sys/kernel/debug/swap/memcg_tiers to set it. Bit i is the tier at index i. /sys/kernel/debug/swap/tiers lists each tier's index and priority.

  # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers

This lives in debugfs on purpose. A cgroup file is a permanent ABI, and what a swap tier should look like is not settled yet. It is built only with CONFIG_MEMCG and CONFIG_DEBUG_FS.

The masks are kept in a list keyed by cgroup ID rather than in struct mem_cgroup, so memcg itself is not changed. A mask applies to the memory charged to its cgroup and is not inherited by child cgroups. The entry of a removed cgroup is dropped on the next write.

Each swap device is stamped with its tier's bit at swapon. A device's tier never changes, so the stamp is written once and read locklessly by the allocator.

When the last device of a tier is swapped off, the tier frees its index for reuse. It waits until swapoff can no longer fail, so a device whose swapoff fails goes back to the same index. The freed bit is set back in every cgroup mask, so a cgroup that had disabled that tier must disable it again once a new tier reuses the index.

Assisted-by Claude:claude-opus-5
---
 Documentation/mm/index.rst     |   1 +
 Documentation/mm/swap-tier.rst |  55 ++++++
 MAINTAINERS                    |   1 +
 include/linux/swap.h           |   1 +
 mm/swap_tier.c                 | 294 ++++++++++++++++++++++++++++++++-
 mm/swap_tier.h                 |  31 ++++
 mm/swapfile.c                  |   5 +
 7 files changed, 387 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/mm/swap-tier.rst
Documentation/mm/index.rst +1 -0
index 13a79f5d092c..6afc45cd4b3d 100644
@@ -34,6 +34,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`.
    page_reclaim
    swap
    swap-table
+   swap-tier
    page_cache
    shmfs
    oom
Documentation/mm/swap-tier.rst new file +55 -0

55 lines not on this page · raw message

MAINTAINERS +1 -0
index 37f353015cae..af17b804f31b 100644
@@ -17402,6 +17402,7 @@ L:	[email protected]
 S:	Maintained
 F:	Documentation/ABI/testing/sysfs-kernel-mm-swap
 F:	Documentation/mm/swap-table.rst
+F:	Documentation/mm/swap-tier.rst
 F:	include/linux/swap.h
 F:	include/linux/swap_ops.h
 F:	include/linux/swapfile.h
include/linux/swap.h +1 -0
index df69c2dd434a..30d3c37ca530 100644
@@ -237,6 +237,7 @@ struct swap_info_struct {
 	struct percpu_ref users;	/* indicate and keep swap device valid. */
 	unsigned long	flags;		/* SWP_USED etc: see above */
 	signed short	prio;		/* swap priority of this type */
+	unsigned int	tier_mask;	/* swap tier mask */
 	struct plist_node list;		/* entry in its swap tier */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
mm/swap_tier.c +293 -1

336 lines not on this page · raw message

mm/swap_tier.h +31 -0
index c4347c28d8f1..9967cfbcc439 100644
@@ -9,16 +9,20 @@
 /* Forward declarations */
 struct swap_info_struct;
 
+#define TIER_ALL_MASK		(~0U)
+
 /*
  * struct swap_tier - structure representing a swap tier.
  *
  * @prio: priority of the swap devices in the tier.
+ * @nr_devs: swap devices in the tier, including ones being swapped off.
  * @active_head: swap devices in the tier.
  * @avail_head: available swap devices in the tier.
  * @list: linkage into swap_tier_active_list or swap_tier_inactive_list.
  */
 struct swap_tier {
 	short prio;
+	int nr_devs;
 	struct plist_head active_head;
 	struct plist_head avail_head;
 	struct list_head list;
@@ -35,6 +39,33 @@ void swap_tiers_init(void);
 /* Tier assignment */
 void swap_tiers_assign_dev(struct swap_info_struct *swp);
 void swap_tiers_remove_dev(struct swap_info_struct *swp);
+unsigned int swap_tiers_release_dev(struct swap_info_struct *swp);
 struct plist_head *swap_tiers_avail_head(struct swap_info_struct *swp);
 
+/**
+ * swap_tiers_mask_test - test whether two tier masks overlap
+ * @tier_mask: mask to test, e.g. a swap device's tier bit
+ * @mask: mask to test against, e.g. a cgroup's mask
+ *
+ * Return: true if @tier_mask and @mask share at least one tier bit.
+ */
+static inline bool swap_tiers_mask_test(unsigned int tier_mask,
+					unsigned int mask)
+{
+	return tier_mask & mask;
+}
+
+#if defined(CONFIG_MEMCG) && defined(CONFIG_DEBUG_FS)
+/* Memcg related functions */
+void swap_tiers_memcg_propagate(unsigned int mask);
+unsigned int folio_tier_mask(struct folio *folio);
+#else
+static inline void swap_tiers_memcg_propagate(unsigned int mask) {}
+
+static inline unsigned int folio_tier_mask(struct folio *folio)
+{
+	return TIER_ALL_MASK;
+}
+#endif
+
 #endif /* _SWAP_TIER_H */
mm/swapfile.c +5 -0
index e9d142c0655b..bb953dd33ca0 100644
@@ -3191,6 +3191,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	struct address_space *mapping;
 	struct inode *inode;
 	unsigned int maxpages;
+	unsigned int freed_tier;
 	int err, found = 0;
 
 	if (!capable(CAP_SYS_ADMIN))
@@ -3290,7 +3291,11 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
 	p->max = 0;
 	p->cluster_info = NULL;
 	spin_unlock(&p->lock);
+	freed_tier = swap_tiers_release_dev(p);
 	spin_unlock(&swap_lock);
+	/* Under swapon_mutex, so a swapon cannot reuse the index before this. */
+	if (freed_tier)
+		swap_tiers_memcg_propagate(freed_tier);
 	arch_swap_invalidate_area(p->type);
 	zswap_swapoff(p->type);
 	mutex_unlock(&swapon_mutex);
Signature · 1 line
2.48.1
author reviewer
#4 lore raw

RFC PATCH v11 4/4 mm: swap: filter swap allocation by memcg tier mask

In reply to Youngjun Park

Apply the cgroup tier mask during swap slot allocation to enforce per-cgroup swap tier restrictions.

The folio's mask is looked up once and passed to the fast, slow and discard paths as a parameter, so all of them act on the same mask even if the cgroup's mask changes concurrently.

The device tier_mask is read with READ_ONCE() in the two paths that do not hold swap_lock, matching how si->flags is read in the same allocator.

In the fast path, check the percpu cached swap_info's tier_mask against the folio's mask. If it does not match, fall through to the slow path. In the slow path, skip swap devices whose tier_mask is not covered by the folio's mask. The discard fallback honors the mask too. Without it, a discard on a device outside the folio's tiers still returns true and drives the retry, so the allocation spins through the loop consuming another device's discard queue while it cannot succeed.

This works correctly when there is only one non-rotational device in the system and no devices share the same priority. However, there are known limitations.

- When non-rotational devices are distributed across multiple tiers, and different memcgs are configured to use those distinct tiers, they may constantly overwrite the shared percpu swap cache. This cache thrashing leads to frequent fast path misses.

- Combined with the above issue, if same-priority devices exist among them, a percpu cache miss (overwritten by another memcg) forces the allocator to round-robin to the next device prematurely, even if the current cluster is not fully exhausted.

These edge cases do not affect the primary use case of directing swap traffic per cgroup. Further optimization is planned for future work.

---
 mm/swapfile.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)
mm/swapfile.c +17 -7
index bb953dd33ca0..b246eff25c96 100644
@@ -1364,7 +1364,7 @@ static bool get_swap_device_info(struct swap_info_struct *si)
  * Fast path try to get swap entries with specified order from current
  * CPU's swap entry pool (a cluster).
  */
-static bool swap_alloc_fast(struct folio *folio)
+static bool swap_alloc_fast(struct folio *folio, unsigned int mask)
 {
 	unsigned int order = folio_order(folio);
 	struct swap_cluster_info *ci;
@@ -1376,8 +1376,11 @@ static bool swap_alloc_fast(struct folio *folio)
 	 * so checking it's liveness by get_swap_device_info is enough.
 	 */
 	si = this_cpu_read(percpu_swap_cluster.si[order]);
+	if (!si || !swap_tiers_mask_test(READ_ONCE(si->tier_mask), mask))
+		return false;
+
 	offset = this_cpu_read(percpu_swap_cluster.offset[order]);
-	if (!si || !offset || !get_swap_device_info(si))
+	if (!offset || !get_swap_device_info(si))
 		return false;
 
 	ci = swap_cluster_lock(si, offset);
@@ -1394,7 +1397,7 @@ static bool swap_alloc_fast(struct folio *folio)
 }
 
 /* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static void swap_alloc_slow(struct folio *folio, unsigned int mask)
 {
 	struct swap_info_struct *si, *next;
 	struct swap_tier *tier;
@@ -1405,6 +1408,9 @@ static void swap_alloc_slow(struct folio *folio)
 	for_each_active_tier(tier) {
 		prio = tier->prio;
 		plist_for_each_entry_safe(si, next, &tier->avail_head, avail_list) {
+			if (!swap_tiers_mask_test(READ_ONCE(si->tier_mask), mask))
+				continue;
+
 			/* Rotate the device and switch to a new cluster */
 			plist_requeue(&si->avail_list, &tier->avail_head);
 			spin_unlock(&swap_avail_lock);
@@ -1439,7 +1445,7 @@ static void swap_alloc_slow(struct folio *folio)
  * Discard pending clusters in a synchronized way when under high pressure.
  * Return: true if any cluster is discarded.
  */
-static bool swap_sync_discard(void)
+static bool swap_sync_discard(unsigned int mask)
 {
 	bool ret = false;
 	struct swap_info_struct *si, *next;
@@ -1451,6 +1457,8 @@ static bool swap_sync_discard(void)
 	for_each_active_tier(tier) {
 		prio = tier->prio;
 		plist_for_each_entry_safe(si, next, &tier->active_head, list) {
+			if (!swap_tiers_mask_test(si->tier_mask, mask))
+				continue;
 			spin_unlock(&swap_lock);
 			if (get_swap_device_info(si)) {
 				if (si->flags & SWP_PAGE_DISCARD)
@@ -1749,6 +1757,7 @@ int folio_alloc_swap(struct folio *folio)
 {
 	unsigned int order = folio_order(folio);
 	unsigned int size = 1 << order;
+	unsigned int mask;
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -1772,13 +1781,14 @@ int folio_alloc_swap(struct folio *folio)
 	}
 
 again:
+	mask = folio_tier_mask(folio);
 	local_lock(&percpu_swap_cluster.lock);
-	if (!swap_alloc_fast(folio))
-		swap_alloc_slow(folio);
+	if (!swap_alloc_fast(folio, mask))
+		swap_alloc_slow(folio, mask);
 	local_unlock(&percpu_swap_cluster.lock);
 
 	if (!order && unlikely(!folio_test_swapcache(folio))) {
-		if (swap_sync_discard())
+		if (swap_sync_discard(mask))
 			goto again;
 	}
 
Signature · 1 line
2.48.1
maintainer
#5 lore raw
In reply to Youngjun Park
Quoted 13 lines · Youngjun Park

Per-cgroup swap in debugfs
==========================

Patches 3 and 4 let a memory cgroup choose its tiers through debugfs.

  # swapon -p 100 /dev/nvme0n1p2
  # swapon -p 50 /dev/sdb2
  # cat /sys/kernel/debug/swap/tiers
  Idx   Prio
  0     100
  1     50
  # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers
Quoted 3 lines

Bit i of the mask is tier i, so /batch swaps only to sdb2. A tier keeps its index for its lifetime, so the mask keeps selecting the same tier across swapon and swapoff.

Can the cgroup be given a priority limit? That would have pretty obvious inheritance semantics:

root
`- batch (memory.swap.prio.max = 20)
`- task (memory.swap.prio.max = max)
`- logs (memory.swap.prio.max = 10)
`- interactive (memory.swap.prio.max = max)
`- task (memory.swap.prio.max)

#6 lore raw
Quoted 14 lines · Johannes Weiner
Quoted 13 lines · Youngjun Park

Per-cgroup swap in debugfs
==========================

Patches 3 and 4 let a memory cgroup choose its tiers through debugfs.

  # swapon -p 100 /dev/nvme0n1p2
  # swapon -p 50 /dev/sdb2
  # cat /sys/kernel/debug/swap/tiers
  Idx   Prio
  0     100
  1     50
  # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers
Quoted 4 lines
Quoted 3 lines

Bit i of the mask is tier i, so /batch swaps only to sdb2. A tier keeps its index for its lifetime, so the mask keeps selecting the same tier across swapon and swapoff.

Hello Johannes,

Sorry for the late reply on a good suggestion :)

Quoted 8 lines

Can the cgroup be given a priority limit? That would have pretty obvious inheritance semantics:
root
`- batch (memory.swap.prio.max = 20)
`- task (memory.swap.prio.max = max)
`- logs (memory.swap.prio.max = 10)
`- interactive (memory.swap.prio.max = max)
`- task (memory.swap.prio.max)

Right, the inheritance is clear and easy to understand, and with this I can pre-define the limit without knowing the mask value.

But first, let me check the intent. Is the point that capping batch keeps it from taking the faster tiers, so they are left for interactive?

If so, that matches our use case. Latency sensitive workloads get the fast tiers, non-latency sensitive ones get the slow tiers. But...

Even then, the reverse cannot be expressed. A cap only cuts from the top, so a latency sensitive workload given max can still fall back to the slow tiers once the fast ones fill up. For example,

tier0 tier1 tier2 tier3

    0     10     20     30

there is no way to say "use tier0 and tier1, but never fall back to tier2 or tier3". To cover that, the interface would also need a min value, or some way to express a range.

And even a range is not enough. Excluding only tier2 leaves a hole in the middle, which no min/max pair can express. That needs per-tier selection, which is what the mask, and what I'd carry over to the memcg interface later (Currently memcg.swap.tiers.max).

How do you think?

Thanks!
Youngjun Park

maintainer
#7 lore raw
Quoted 15 lines · Youngjun Park
Quoted 14 lines · Johannes Weiner
Quoted 13 lines · Youngjun Park

Per-cgroup swap in debugfs
==========================

Patches 3 and 4 let a memory cgroup choose its tiers through debugfs.

  # swapon -p 100 /dev/nvme0n1p2
  # swapon -p 50 /dev/sdb2
  # cat /sys/kernel/debug/swap/tiers
  Idx   Prio
  0     100
  1     50
  # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Bit i of the mask is tier i, so /batch swaps only to sdb2. A tier keeps its index for its lifetime, so the mask keeps selecting the same tier across swapon and swapoff.

Hello Johannes,

Sorry for the late reply on a good suggestion :)

No worries, and same ^_^

Quoted 9 lines
Quoted 8 lines

Can the cgroup be given a priority limit? That would have pretty obvious inheritance semantics:
root
`- batch (memory.swap.prio.max = 20)
`- task (memory.swap.prio.max = max)
`- logs (memory.swap.prio.max = 10)
`- interactive (memory.swap.prio.max = max)
`- task (memory.swap.prio.max)

Quoted 5 lines

Right, the inheritance is clear and easy to understand, and with this I can pre-define the limit without knowing the mask value.

But first, let me check the intent. Is the point that capping batch keeps it from taking the faster tiers, so they are left for interactive?

Yes, basically, that's what I tried to express. Interactive has access to all available capacity. Batch only has access to lower tiers.

Quoted 7 lines

If so, that matches our use case. Latency sensitive workloads get the fast tiers, non-latency sensitive ones get the slow tiers. But...

Even then, the reverse cannot be expressed. A cap only cuts from the top, so a latency sensitive workload given max can still fall back to the slow tiers once the fast ones fill up. For example,

Quoted 6 lines

tier0 tier1 tier2 tier3

    0     10     20     30

there is no way to say "use tier0 and tier1, but never fall back to tier2 or tier3". To cover that, the interface would also need a min value, or some way to express a range.

Correct, this isn't covered by the above.

Quoted 6 lines

And even a range is not enough. Excluding only tier2 leaves a hole in the middle, which no min/max pair can express. That needs per-tier selection, which is what the mask, and what I'd carry over to the memcg interface later (Currently memcg.swap.tiers.max).

How do you think?

I think it could help to aggregate the usecases in the cover letter. Your cover letter describes how it works, which is great, but it would be good to understand better what the constraints are, how it fits in with other existing control surface and broader usage models.

With the above, yes, you can restrict who gets access to the privileged tiers top down, but not bottom up. Is that an issue? Keep in mind the alternative is cutting privileged groups OFF from certain available capacity. This seems somewhat counter-intuitive to me, and doesn't reflect a clean privilege hierarchy anymore.

If you can think of a good usecase, memory.swap.prio.min would be certainly a natural extension. But we should get the usecase laid out.

The requirement to punch holes is the one I can relate to least. Why would a cgroup need access to good tiers and bad tiers, but skip the middle ones?

This would seem less like tiering/hierarchy and more like flat per-cgroup swap pools but with obstacles.

#8 lore raw
Quoted 9 lines · Johannes Weiner

If you can think of a good usecase, memory.swap.prio.min would be certainly a natural extension. But we should get the usecase laid out.

The requirement to punch holes is the one I can relate to least. Why would a cgroup need access to good tiers and bad tiers, but skip the middle ones?

This would seem less like tiering/hierarchy and more like flat per-cgroup swap pools but with obstacles.

Hi Johannes and Youngjun,

Sorry that I am only joining this part of the discussion now.

I recently started helping carry Kairui's swap queue work forward. Our current v2 is here [1]. Youngjun's tier work interacts directly with it:
v2 maintains a device queue and reader for each priority, while v11 makes each priority a tier and moves device selection under that tier.

I did a functional integration test of v11 on an x86-64 host. The tier mask, fallback, same-priority device allocation, concurrent allocation, and swapoff smoke tests all worked without kernel warnings. This was a zram integration test, not yet a real multi-SSD performance test.

One result seems relevant to this discussion: with a restricted parent and an unconfigured child, the child could still use the faster tier. I therefore agree that an inheritable memory.swap.prio.max looks like the cleaner first cgroup interface. A hole in the mask worked mechanically, but I do not yet have a convincing hierarchical use case for it; it felt more like per-cgroup swap-pool membership.

For the queue integration, one possible boundary is for the tier to own the queue/reader and for the swap queue to become the default per-tier device allocation policy. I would like to align this with both of you before changing v2. I will also continue the real multi-SSD tests.

If either of you has suggestions or specific test cases you would like to see, please let me know. I have a few test setups available and should be able to try some of them. I am still getting up to speed on this part of MM, so please correct me if I missed some context or got any detail wrong.

[1] https://lore.kernel.org/all/[email protected]/

Thanks,
Lian

#9 lore raw

Hi Lian!

Quoted 4 lines

zram integration test, not yet a real multi-SSD performance test.

One result seems relevant to this discussion: with a restricted parent and an unconfigured child, the child could still use the faster tier. I

I intentionally left out the parent/child hierarchy handling in this version, since I thought it would be needed only at memcg integration time. At this point, though, I think it is better to keep the hierarchy even in the debugfs interface,
so after reconsideration. I'll do that in v12!

Quoted 4 lines

therefore agree that an inheritable memory.swap.prio.max looks like the cleaner first cgroup interface. A hole in the mask worked mechanically, but I do not yet have a convincing hierarchical use case for it; it felt more like per-cgroup swap-pool membership.

I plan to proceed with v12 as Johannes suggested. (I've replied with my thoughts in the thread.)

Quoted 4 lines

For the queue integration, one possible boundary is for the tier to own the queue/reader and for the swap queue to become the default per-tier device allocation policy. I would like to align this with both of you before changing v2. I will also continue the real multi-SSD tests.

I'd like to hear what Kairui and Chris think as well. If they agree, this is the direction I prefer. Honestly, beyond preference, I think it is the right one :) instead of allocating a separate swap queue structure, the swap tier itself can be used for it!

Quoted 5 lines

If either of you has suggestions or specific test cases you would like to see, please let me know. I have a few test setups available and should be able to try some of them. I am still getting up to speed on this part of MM, so please correct me if I missed some context or got any detail wrong.

When I send v12, I'll spell out more clearly the parts where we can collaborate and what I'd like to propose to you. If anything comes up before then, I'll reply on the v2 thread.

Thanks,
Youngjun

#10 lore raw
In reply to Johannes Weiner
Quoted 72 lines · Johannes Weiner
Quoted 15 lines · Youngjun Park
Quoted 14 lines · Johannes Weiner
Quoted 13 lines · Youngjun Park

Per-cgroup swap in debugfs
==========================

Patches 3 and 4 let a memory cgroup choose its tiers through debugfs.

  # swapon -p 100 /dev/nvme0n1p2
  # swapon -p 50 /dev/sdb2
  # cat /sys/kernel/debug/swap/tiers
  Idx   Prio
  0     100
  1     50
  # echo "/batch 0x2" > /sys/kernel/debug/swap/memcg_tiers
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Bit i of the mask is tier i, so /batch swaps only to sdb2. A tier keeps its index for its lifetime, so the mask keeps selecting the same tier across swapon and swapoff.

Hello Johannes,

Sorry for the late reply on a good suggestion :)

No worries, and same ^_^

Quoted 9 lines
Quoted 8 lines

Can the cgroup be given a priority limit? That would have pretty obvious inheritance semantics:
root
`- batch (memory.swap.prio.max = 20)
`- task (memory.swap.prio.max = max)
`- logs (memory.swap.prio.max = 10)
`- interactive (memory.swap.prio.max = max)
`- task (memory.swap.prio.max)

Quoted 5 lines

Right, the inheritance is clear and easy to understand, and with this I can pre-define the limit without knowing the mask value.

But first, let me check the intent. Is the point that capping batch keeps it from taking the faster tiers, so they are left for interactive?

Yes, basically, that's what I tried to express. Interactive has access to all available capacity. Batch only has access to lower tiers.

Quoted 7 lines

If so, that matches our use case. Latency sensitive workloads get the fast tiers, non-latency sensitive ones get the slow tiers. But...

Even then, the reverse cannot be expressed. A cap only cuts from the top, so a latency sensitive workload given max can still fall back to the slow tiers once the fast ones fill up. For example,

Quoted 6 lines

tier0 tier1 tier2 tier3

    0     10     20     30

there is no way to say "use tier0 and tier1, but never fall back to tier2 or tier3". To cover that, the interface would also need a min value, or some way to express a range.

Correct, this isn't covered by the above.

Quoted 6 lines

And even a range is not enough. Excluding only tier2 leaves a hole in the middle, which no min/max pair can express. That needs per-tier selection, which is what the mask, and what I'd carry over to the memcg interface later (Currently memcg.swap.tiers.max).

How do you think?

I think it could help to aggregate the usecases in the cover letter. Your cover letter describes how it works, which is great, but it would be good to understand better what the constraints are, how it fits in with other existing control surface and broader usage models.

Quoted 5 lines

With the above, yes, you can restrict who gets access to the privileged tiers top down, but not bottom up. Is that an issue? Keep in mind the alternative is cutting privileged groups OFF from certain available capacity. This seems somewhat counter-intuitive to me, and doesn't reflect a clean privilege hierarchy anymore.

Thanks for the explanation.

I thought about it many times and concluded you are right. The reason is simple. the two cases the mask can express but prio.max cannot (holes and bottom-up restriction) are possible with the mask, but there is no real use case that needs them.

Therefore I see no reason to keep the
mask interface I proposed. prio.max can be extended naturally when needed.

For the record, the interface semantics have evolved as below, and this discussion changes them once more.

1. Per-cgroup swap priority for our usecase (initial proposal)
2. Fixed ordering across cgroups (swap tier concept introduced)
3. Child tiers as a subset of the parent's (after LPC)
4. Switch to the memory.swap.tier.max interface
5. Agreement to add the memcg interface after virtual swap lands
6. Express tiers by swap priority, proposed as a debugfs interface until the memcg interface is added
7. memory.swap.prio.max: cap the fastest tier a cgroup may use, i.e. top-down restriction (this discussion)

I agree with this direction. I'll describe the discussion history and the resulting constraints more clearly in the cover letter.

Here are the next steps.

- Change the debugfs interface to take a maximum priority instead of a tier mask.
- Introduce memory.swap.prio.max after virtual swap lands.

Please let me know if you have further comments.

Thanks,
Youngjun