Skip to contentSkip to conversation

Reclaimable kernel stacks

RFC In review David Stevens 10 participants 54 messages Aug 27 – Sep 1

Subsystems (14)
Memory management x86 VFS MM core +10 more subsystems
lore
Share & tools
lore
#0 lore raw

cover Reclaimable kernel stacks

This RFC is a different approach to reducing kernel stack usage from the earlier dynamic kernel stack RFC [1]. This patch series aims to reduce the cost of kernel stacks by partially reclaiming stacks of blocked tasks when it is safe to do so.

On Android, system processes typically have 2000-3000 threads. App processes add 1000s more threads on top of this. The number of app processes varies based on device RAM size, but the end result is that 1-2% of system RAM is consumed by kernel stacks. However, most of these threads spend extended periods of time blocked. As such, reclaiming blocked kernel stacks can reduce total kernel stack memory usage by upwards of 50% in various multi-tasking test cases.

When a task is blocked, we know exactly where the top of its stack is and can reclaim any pages past that point. Since any accesses to that portion of the stack are bugs like use-after-return or buffer overflow, turning those invalid accesses into hard crashes could even be considered a positive.

Tracking blocked state and when it is safe to reclaim a stack is done via a series of hooks in the scheduler. The actual reclaim of stacks is done asynchronously in a shrinker.

Once a task's stack has been reclaimed, it cannot be rescheduled until its stack is repopulated. Although there can be a repopulation fast path within the scheduler, reliably allocating memory to repopulate the stack requires a fallback path that defers the repopulation and wakeup to a workqueue context that can use GFP_KERNEL.

The primary challenge is avoiding reclaim deadlocks. If a task blocks while holding a lock used by direct reclaim and then has its stack reclaimed, using GFP_KERNEL to reallocate its stack risks deadlock. To avoid this, only tasks which are known not to hold any locks upon which reclaim depends are considered eligible for stack reclaim. Automatically inferring this property is not feasible, so instead a new PF_RECLAIMABLE_STACK task flag is used to annotate blocking locations that are known safe. While annotating all safe blocking locations is not feasible, the vast majority of userspace threads block using a fairly small number of syscalls - futex, epoll, nanosleep, etc. The 10 annotations added in this series cover >95% of userspace threads on Android based on my testing. Since missing annotations are leaving an optimization on the table rather than an actual bug, other annotations can be added later as needed.

Although reclaimable stacks will not cause reclaim to deadlock, it does introduce a dependency on needing to allocate memory before an OOM victim can exit, as reclaimed stacks need to be repopulated before their tasks can run. While the OOM reaper will still be able to immediately free the victim's mm, the freeing of non-mm memory may be delayed. This can lead to more OOM kills. While this is not a significant concern on Android due to the reliance on lmkd over the kernel OOM killer, it may be a concern on other systems.

This RFC was developed primarily on 6.18 and 7.1 based kernels. I have done fairly heavy stress testing, but it has not yet been deployed to any production systems. If initial feedback on the RFC is somewhat positive, I will work on deploying it to production systems for further stability and performance testing as well as resolving the handful of TODOs left in the RFC.

[1] https://lore.kernel.org/linux-mm/[email protected]/

David Stevens (10):
Add !MEMCG memcg_list_lru_alloc implementation mm/vmalloc: Skip vmallocinfo NUMA stats for VM_SPARSE
fork: refactor vmap stack alloc/free into helpers
mm: vmalloc: support creating aligned vm areas
fork: allocate reclaimable stacks with VM_SPARSE Reclaim memory from blocked kernel stacks Reclaim stacks via a shrinker
Set PF_RECLAIMABLE_STACK in various places
x86: Enable reclaimable stacks
arm64: Enable reclaimable stacks

 arch/Kconfig                       |  18 +
 arch/arm64/Kconfig                 |   1 +
 arch/arm64/include/asm/processor.h |   5 +
 arch/x86/Kconfig                   |   1 +
 arch/x86/include/asm/processor.h   |   5 +
 drivers/android/binder/thread.rs   |  14 +
 fs/eventpoll.c                     |   3 +
 fs/pipe.c                          |  28 +-
 fs/select.c                        |   3 +
 include/linux/list_lru.h           |   7 +-
 include/linux/sched.h              |  44 +-
 include/linux/sched/task_stack.h   |  23 +
 include/linux/vmalloc.h            |   2 +
 kernel/Makefile                    |   2 +
 kernel/fork.c                      | 140 +++++-
 kernel/futex/waitwake.c            |   3 +
 kernel/sched/core.c                |  18 +-
 kernel/sched/sched.h               |   3 +
 kernel/signal.c                    |  36 +-
 kernel/stack_shrinker.c            | 760 +++++++++++++++++++++++++++++
 kernel/stack_shrinker.h            |  58 +++
 kernel/time/hrtimer.c              |   3 +
 mm/vmalloc.c                       |  27 +-
 rust/kernel/task.rs                |  16 +
 24 files changed, 1175 insertions(+), 45 deletions(-)
 create mode 100644 kernel/stack_shrinker.c
 create mode 100644 kernel/stack_shrinker.h

base-commit: 8d3ae59288f1e7d58d76558a6ee96d533bc5019f

Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#1 lore raw

RFC 1/10 Add !MEMCG memcg_list_lru_alloc implementation

Add a stub implementation of memcg_list_lru_alloc() when CONFIG_MEMCG isn't enabled.

---
 include/linux/list_lru.h | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
include/linux/list_lru.h +6 -1
index a450fffe1550..40eaeaa3424c 100644
@@ -79,10 +79,10 @@ static inline int list_lru_init_memcg_key(struct list_lru *lru, struct shrinker
 	return list_lru_init_memcg(lru, shrinker);
 }
 
+#ifdef CONFIG_MEMCG
 int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
 			 gfp_t gfp);
 
-#ifdef CONFIG_MEMCG
 /**
  * folio_memcg_list_lru_alloc - allocate list_lru heads for shrinkable folio
  * @folio: the newly allocated & charged folio
@@ -106,6 +106,11 @@ static inline int folio_memcg_list_lru_alloc(struct folio *folio,
 {
 	return 0;
 }
+
+static inline int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp)
+{
+	return 0;
+}
 #endif
 
 void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#2 lore raw

RFC 2/10 mm/vmalloc: Skip vmallocinfo NUMA stats for VM_SPARSE

In reply to David Stevens

Bail from show_numa_info() in vmalloc_info_show() for areas with VM_SPARSE set. This lets clients that allocate VM_SPARSE areas directly use vm_struct's nr_pages and pages fields.

---
 mm/vmalloc.c | 3 +++
 1 file changed, 3 insertions(+)
mm/vmalloc.c +3 -0
index f4fa227a8d7f..d39897300f29 100644
@@ -5294,6 +5294,9 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v,
 	if (!counters)
 		return;
 
+	if (v->flags & VM_SPARSE)
+		return;
+
 	memset(counters, 0, nr_node_ids * sizeof(unsigned int));
 
 	for (nr = 0; nr < v->nr_pages; nr += step)
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#3 lore raw

RFC 3/10 fork: refactor vmap stack alloc/free into helpers

In reply to David Stevens

Prepare for alternate implementations of some functions for reclaimable stacks by creating helpers for allocating and freeing vmap stacks. Also reorder some functions to consolidate the functions that will have alternate implementations.

---
 kernel/fork.c | 41 +++++++++++++++++++++++++++--------------
 1 file changed, 27 insertions(+), 14 deletions(-)
kernel/fork.c +27 -14
index f0e2e131a9a5..10bd76f6dd20 100644
@@ -207,11 +207,6 @@ static DEFINE_PER_CPU(struct vm_struct *, cached_stacks[NR_CACHED_STACKS]);
  */
 #define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)
 
-struct vm_stack {
-	struct rcu_head rcu;
-	struct vm_struct *stack_vm_area;
-};
-
 static struct vm_struct *alloc_thread_stack_node_from_cache(struct task_struct *tsk, int node)
 {
 	struct vm_struct *vm_area;
@@ -272,6 +267,26 @@ static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area)
 	return false;
 }
 
+static void free_vmap_stack(struct vm_struct *vm_area)
+{
+	vfree(vm_area->addr);
+}
+
+static struct vm_struct *alloc_vmap_stack(int node)
+{
+	void *stack = __vmalloc_node(THREAD_SIZE, THREAD_ALIGN,
+				     GFP_VMAP_STACK,
+				     node, __builtin_return_address(0));
+	if (!stack)
+		return NULL;
+	return find_vm_area(stack);
+}
+
+struct vm_stack {
+	struct rcu_head rcu;
+	struct vm_struct *stack_vm_area;
+};
+
 static void thread_stack_free_rcu(struct rcu_head *rh)
 {
 	struct vm_stack *vm_stack = container_of(rh, struct vm_stack, rcu);
@@ -280,7 +295,7 @@ static void thread_stack_free_rcu(struct rcu_head *rh)
 	if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area))
 		return;
 
-	vfree(vm_area->addr);
+	free_vmap_stack(vm_area);
 }
 
 static void thread_stack_delayed_free(struct task_struct *tsk)
@@ -302,7 +317,7 @@ static int free_vm_stack_cache(unsigned int cpu)
 		if (!vm_area)
 			continue;
 
-		vfree(vm_area->addr);
+		free_vmap_stack(vm_area);
 		cached_vm_stack_areas[i] = NULL;
 	}
 
@@ -338,7 +353,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node)
 	vm_area = alloc_thread_stack_node_from_cache(tsk, node);
 	if (vm_area) {
 		if (memcg_charge_kernel_stack(vm_area)) {
-			vfree(vm_area->addr);
+			free_vmap_stack(vm_area);
 			return -ENOMEM;
 		}
 
@@ -356,15 +371,13 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node)
 		return 0;
 	}
 
-	stack = __vmalloc_node(THREAD_SIZE, THREAD_ALIGN,
-				     GFP_VMAP_STACK,
-				     node, __builtin_return_address(0));
-	if (!stack)
+	vm_area = alloc_vmap_stack(node);
+	if (!vm_area)
 		return -ENOMEM;
+	stack = vm_area->addr;
 
-	vm_area = find_vm_area(stack);
 	if (memcg_charge_kernel_stack(vm_area)) {
-		vfree(stack);
+		free_vmap_stack(vm_area);
 		return -ENOMEM;
 	}
 	/*
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#4 lore raw

RFC 4/10 mm: vmalloc: support creating aligned vm areas

In reply to David Stevens

Expose a get_vm_area_node() function that supports passing an alignment when allocating a vm area.

---
 include/linux/vmalloc.h |  2 ++
 mm/vmalloc.c            | 24 +++++++++++++++++++++++-
 2 files changed, 25 insertions(+), 1 deletion(-)
include/linux/vmalloc.h +2 -0
index d87dc7f77f4e..5ed9c5b25026 100644
@@ -246,6 +246,8 @@ static inline size_t get_vm_area_size(const struct vm_struct *area)
 extern struct vm_struct *get_vm_area(unsigned long size, unsigned long flags);
 extern struct vm_struct *get_vm_area_caller(unsigned long size,
 					unsigned long flags, const void *caller);
+extern struct vm_struct *get_vm_area_node(unsigned long size, unsigned long align,
+					  unsigned long flags, int node);
 extern struct vm_struct *__get_vm_area_caller(unsigned long size,
 					unsigned long flags,
 					unsigned long start, unsigned long end,
mm/vmalloc.c +23 -1
index d39897300f29..059cd0748514 100644
@@ -3280,7 +3280,7 @@ struct vm_struct *__get_vm_area_caller(unsigned long size, unsigned long flags,
  * @flags:	 %VM_IOREMAP for I/O mappings or VM_ALLOC
  *
  * Search an area of @size in the kernel virtual mapping area,
- * and reserved it for out purposes.  Returns the area descriptor
+ * and reserved it for our purposes.  Returns the area descriptor
  * on success or %NULL on failure.
  *
  * Return: the area descriptor on success or %NULL on failure.
@@ -3301,6 +3301,28 @@ struct vm_struct *get_vm_area_caller(unsigned long size, unsigned long flags,
 				  NUMA_NO_NODE, GFP_KERNEL, caller);
 }
 
+/**
+ * get_vm_area_node - reserve a contiguous kernel virtual area
+ * @size:	 size of the area
+ * @align:	 alignment of the area
+ * @flags:	 %VM_IOREMAP for I/O mappings or VM_ALLOC
+ * @node:	 node from which to allocate data structures
+ *
+ * Search an area of @size in the kernel virtual mapping area,
+ * and reserve it for our purposes.  Returns the area descriptor
+ * on success or %NULL on failure.
+ *
+ * Return: the area descriptor on success or %NULL on failure.
+ */
+struct vm_struct *get_vm_area_node(unsigned long size, unsigned long align,
+				   unsigned long flags, int node)
+{
+	return __get_vm_area_node(size, align, PAGE_SHIFT, flags,
+				  VMALLOC_START, VMALLOC_END,
+				  node, GFP_KERNEL,
+				  __builtin_return_address(0));
+}
+
 /**
  * find_vm_area - find a continuous kernel virtual area
  * @addr:	  base address
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#5 lore raw

RFC 5/10 fork: allocate reclaimable stacks with VM_SPARSE

In reply to David Stevens

Since the pages of reclaimable stacks will not always be populated, we need to set VM_SPARSE on their vm_structs to avoid crashes when vread_iter sees a partially reclaimed stack. Note that stacks will only be partially reclaimed when they are associated with a live task, so the stack management and caching code in fork.c won't actually ever see a partially reclaimed stack.

Directly managing the vm_structs instead of going through vmalloc also requires doing the freeing of the stack on a work queue instead of in an RCU callback. Previously, we were relying on deferred vfree work to move much of the cleanup work from softirq to process context.

The fact that vmap stack pages are included in vmalloc's vmstat is well known. We should continue that to avoid changing procfs.

---
 kernel/fork.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
kernel/fork.c +92 -0
index 10bd76f6dd20..6acad0038b78 100644
@@ -267,6 +267,97 @@ static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area)
 	return false;
 }
 
+#ifdef CONFIG_RECLAIMABLE_STACK
+static void free_vmap_stack(struct vm_struct *vm_area)
+{
+	int i;
+
+	remove_vm_area(vm_area->addr);
+
+	for (i = 0; i < vm_area->nr_pages; i++) {
+		mod_node_page_state(page_pgdat(vm_area->pages[i]), NR_VMALLOC, -1);
+		__free_page(vm_area->pages[i]);
+	}
+
+	kfree(vm_area->pages);
+	kfree(vm_area);
+}
+
+static struct vm_struct *alloc_vmap_stack(int node)
+{
+	struct vm_struct *vm_area;
+	int ret;
+
+	vm_area = get_vm_area_node(THREAD_SIZE, THREAD_ALIGN, VM_MAP | VM_SPARSE, node);
+	if (!vm_area)
+		return NULL;
+
+	vm_area->pages = kcalloc_node(THREAD_SIZE >> PAGE_SHIFT, sizeof(*vm_area->pages),
+				      GFP_KERNEL | __GFP_ZERO, node);
+	if (!vm_area->pages)
+		goto alloc_failure;
+
+	while (vm_area->nr_pages < THREAD_SIZE >> PAGE_SHIFT) {
+		struct page *page;
+		gfp_t gfp = GFP_VMAP_STACK | __GFP_HIGHMEM;
+
+		if (node == NUMA_NO_NODE)
+			page = alloc_pages(gfp, 0);
+		else
+			page = alloc_pages_node(node, gfp, 0);
+
+		if (!page)
+			goto alloc_failure;
+
+		/*
+		 * Non-reclaimable vmap stacks pages aren't charged against an
+		 * memcg until account_kernel_stack(), but they are added to
+		 * the node's NR_VMALLOC counter. Copy that behavior to avoid
+		 * confusing userspace.
+		 */
+		mod_node_page_state(page_pgdat(page), NR_VMALLOC, 1);
+		vm_area->pages[vm_area->nr_pages++] = page;
+	}
+
+	ret = vmap_pages_range((unsigned long)vm_area->addr,
+			       (unsigned long)vm_area->addr + THREAD_SIZE,
+			       PAGE_KERNEL, vm_area->pages, PAGE_SHIFT);
+	if (ret)
+		goto alloc_failure;
+
+	return vm_area;
+
+alloc_failure:
+	free_vmap_stack(vm_area);
+	return NULL;
+}
+
+struct vm_stack {
+	struct rcu_work work;
+	struct vm_struct *stack_vm_area;
+};
+
+static void thread_stack_free_work(struct work_struct *work)
+{
+	struct vm_stack *vm_stack = container_of(to_rcu_work(work), struct vm_stack, work);
+	struct vm_struct *vm_area = vm_stack->stack_vm_area;
+
+	if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area))
+		return;
+
+	free_vmap_stack(vm_area);
+}
+
+static void thread_stack_delayed_free(struct task_struct *tsk)
+{
+	struct vm_stack *vm_stack = tsk->stack;
+
+	vm_stack->stack_vm_area = tsk->stack_vm_area;
+	INIT_RCU_WORK(&vm_stack->work, thread_stack_free_work);
+	queue_rcu_work(system_wq, &vm_stack->work);
+}
+
+#else /* !CONFIG_RECLAIMABLE_STACK */
 static void free_vmap_stack(struct vm_struct *vm_area)
 {
 	vfree(vm_area->addr);
@@ -305,6 +396,7 @@ static void thread_stack_delayed_free(struct task_struct *tsk)
 	vm_stack->stack_vm_area = tsk->stack_vm_area;
 	call_rcu(&vm_stack->rcu, thread_stack_free_rcu);
 }
+#endif /* CONFIG_RECLAIMABLE_STACK */
 
 static int free_vm_stack_cache(unsigned int cpu)
 {
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#6 lore raw

RFC 6/10 Reclaim memory from blocked kernel stacks

In reply to David Stevens

Although an individual kernel stack is cheap, the cost can add up on thread heavy systems. On such systems, many threads are often blocked for extended periods of time, waiting for userspace or external events. When a task is blocked, its stack is in a stable, well-defined state. Since we can tell exactly which part of those stacks are unused, we can free the unused portions and reduce the global memory footprint of kernel stacks.

That said, not all blocked tasks can safely have their stacks reclaimed. Since a running task may need all of its stack, a blocked task whose stack has been partially freed cannot be rescheduled until its stack is fully repopulated. Since allocating that memory may require direct reclaim, if a task holding a lock that direct reclaim depends on has its stack reclaimed, we would run the risk of deadlock.

Knowing when tasks hold locks that direct reclaim may depend on is not viable in production systems, especially without lockdep. But noting that most userspace threads will be blocked on events external to the kernel (e.g. waiting on a futex for a userspace controlled wakeup, waiting in epoll for a network packet, waiting on a timer for time to pass, etc), it is possible to enumerate a relatively small number of call sites where most userspace threads will be blocked while holding zero kernel locks.

This change adds a new component that reclaims task stacks. A set of scheduler hooks are used to determine when it is safe to reclaim stacks and to ensure that a task's stack is fully repopulated before being rescheduled. A new PF_RECLAIMABLE_STACK task flag is added that will be used to annotate such safe-to-reclaim call sites. Users of the flag must ensure the task doesn't block while holding a lock while the flag is set. A new TASK_STACK_RECLAIM state is introduced for tasks that are blocked waiting for the stacks to be repopulated.

While reclaimable stacks will not cause direct reclaim to deadlock, it does introduce a dependency on allocating memory before an OOM victim can exit, since any reclaimed stacks need to be repopulated before their threads can run again. This dependency can lead to further OOM kills or potentially an OOM panic if no additional victims are available.

Reclaimable stacks is built on VMAP_STACK. There is no fundamental dependency on !STACK_GROWSUP, but since the only STACK_GROWSUP architecture doesn't support VMAP_STACK, support for STACK_GROWSUP is not implemented.

This feature does work on PREEMPT_RT, but is likely undesirable due to the extra uncertainty. The fact that alloc_pages_nolock_noprof() cannot be called from under the scheduler lock also makes repopulating stacks more expensive.

Enabling reclaimable stacks is mutually exclusive with enabling DEBUG_STACK_USAGE. Making stack_not_used() safe (including from NMIs) may be technically feasible. Even then, the number that it would report would be subtly different: max since last reclaim vs max ever used. This difference seems likely to confuse anything consuming the data, so disabling the feature seems prudent.

Since kernel stacks are not visible in PROC_KCORE when reclaimable stacks is enabled, it is disabled by default when that config is enabled. If a user wants to enable the manually enable the feature and give up visibility, they can make that choice.

In the followup patch, reclaiming task stacks will be done in a shrinker. However, this patch does the reclaim using per-task work structs for simplicity.

---
 arch/Kconfig                     |  18 ++
 include/linux/sched.h            |  39 ++-
 include/linux/sched/task_stack.h |  23 ++
 kernel/Makefile                  |   2 +
 kernel/fork.c                    |   7 +
 kernel/sched/core.c              |  18 +-
 kernel/sched/sched.h             |   3 +
 kernel/stack_shrinker.c          | 464 +++++++++++++++++++++++++++++++
 kernel/stack_shrinker.h          |  58 ++++
 9 files changed, 627 insertions(+), 5 deletions(-)
 create mode 100644 kernel/stack_shrinker.c
 create mode 100644 kernel/stack_shrinker.h
arch/Kconfig +18 -0
index fa7507ac8e13..adb4a5957996 100644
@@ -1534,6 +1534,24 @@ config VMAP_STACK
 	  backing virtual mappings with real shadow memory, and KASAN_VMALLOC
 	  must be enabled.
 
+config HAVE_ARCH_RECLAIMABLE_STACK
+	def_bool n
+
+config RECLAIMABLE_STACK
+	default !PREEMPT_RT && !PROC_KCORE
+	bool "Allow stacks of some blocked threads to be reclaimed"
+	depends on VMAP_STACK && !STACK_GROWSUP
+	depends on HAVE_ARCH_RECLAIMABLE_STACK
+	depends on !DEBUG_STACK_USAGE
+	depends on !KASAN_VMALLOC # TODO: add support for this
+	depends on !DEBUG_KMEMLEAK # TODO: add support for this
+	help
+	  Enable this to allow the unused portion of kernel stacks of most
+	  blocked tasks to be reclaimed.
+
+	  The wakeup latency of tasks with reclaimed stacks may increase,
+	  especially while the system is under memory pressure.
+
 config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET
 	def_bool n
 	help
include/linux/sched.h +37 -2
index 373bcc0598d1..c93a234fac96 100644
@@ -83,6 +83,7 @@ struct sched_dl_entity;
 struct seq_file;
 struct sighand_struct;
 struct signal_struct;
+struct stack_reclaim_work;
 struct task_delay_info;
 struct task_exec_state;
 struct task_group;
@@ -124,7 +125,8 @@ struct user_event_mm;
 #define TASK_FREEZABLE			0x00002000
 #define __TASK_FREEZABLE_UNSAFE	       (0x00004000 * IS_ENABLED(CONFIG_LOCKDEP))
 #define TASK_FROZEN			0x00008000
-#define TASK_STATE_MAX			0x00010000
+#define TASK_STACK_RECLAIM		0x00010000
+#define TASK_STATE_MAX			0x00020000
 
 #define TASK_ANY			(TASK_STATE_MAX-1)
 
@@ -823,6 +825,29 @@ struct kmap_ctrl {
 #endif
 };
 
+#ifdef CONFIG_RECLAIMABLE_STACK
+enum stack_reclaim_enum {
+	STACK_IN_USE		= 0,
+	STACK_PREPARE_RECLAIM	= 1,
+	STACK_RECLAIMABLE	= 2,
+	STACK_RECLAIMING	= 3,
+	STACK_RECLAIMING_IN_USE = 4,
+	STACK_RECLAIMED		= 5,
+} __packed;
+
+union stack_reclaim_state {
+	struct {
+		enum stack_reclaim_enum stack_state;
+		u16 node;
+	};
+	u32 val;
+};
+
+union stack_reclaim_list {
+	struct llist_node refill_entry;
+};
+#endif
+
 struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/*
@@ -1587,6 +1612,16 @@ struct task_struct {
 #endif
 #ifdef CONFIG_VMAP_STACK
 	struct vm_struct		*stack_vm_area;
+#ifdef CONFIG_RECLAIMABLE_STACK
+	union stack_reclaim_state	stack_reclaim_state;
+	union stack_reclaim_list	stack_reclaim_list;
+#ifdef CONFIG_MEMCG
+	struct obj_cgroup		*stack_obj_cgroup;
+#endif
+
+	// TODO: Replace these with a shrinker
+	struct stack_reclaim_work	*stack_reclaim_work;
+#endif
 #endif
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* A live task holds one reference: */
@@ -1796,7 +1831,7 @@ extern struct pid *cad_pid;
 						 * I am cleaning dirty pages from some other bdi. */
 #define PF_KTHREAD		0x00200000	/* I am a kernel thread */
 #define PF_RANDOMIZE		0x00400000	/* Randomize virtual address space */
-#define PF__HOLE__00800000	0x00800000
+#define PF_RECLAIMABLE_STACK	0x00800000	/* This task's stack is reclaimable */
 #define PF__HOLE__01000000	0x01000000
 #define PF__HOLE__02000000	0x02000000
 #define PF_NO_SETAFFINITY	0x04000000	/* Userland is not allowed to meddle with cpus_mask */
include/linux/sched/task_stack.h +23 -0
index 1fab7e9043a3..5d5567899d51 100644
@@ -6,6 +6,7 @@
  * task->stack (kernel stack) handling interfaces:
  */
 
+#include <linux/cleanup.h>
 #include <linux/sched.h>
 #include <linux/magic.h>
 #include <linux/refcount.h>
@@ -83,6 +84,10 @@ static inline void put_task_stack(struct task_struct *tsk) {}
 
 void exit_task_stack_account(struct task_struct *tsk);
 
+/*
+ * Must only be called on current or from inside __schedule() on
+ * prev, to avoid crashing when CONFIG_RECLAIM_STACK is enabled.
+ */
 #define task_stack_end_corrupted(task) \
 		(*(end_of_stack(task)) != STACK_END_MAGIC)
 
@@ -114,4 +119,22 @@ static inline int kstack_end(void *addr)
 	return !(((unsigned long)addr+sizeof(void*)-1) & (THREAD_SIZE-sizeof(void*)));
 }
 
+#ifdef CONFIG_RECLAIMABLE_STACK
+
+DEFINE_CLASS(allow_stack_reclaim, bool,
+	     ({
+		if (!_T)
+			current->flags &= ~PF_RECLAIMABLE_STACK;
+	      }),
+	     ({
+		bool was_set = current->flags & PF_RECLAIMABLE_STACK;
+
+		current->flags |= PF_RECLAIMABLE_STACK;
+		was_set;
+	      }),
+	     void)
+#else /* !CONFIG_RECLAIMABLE_STACK */
+DEFINE_CLASS(allow_stack_reclaim, bool, ({ (void)_T; }), ({ false; }), void)
+#endif /* !CONFIG_VMAP_STACK */
+
 #endif /* _LINUX_SCHED_TASK_STACK_H */
kernel/Makefile +2 -0
index 1e1a31673577..01547102e13a 100644
@@ -12,6 +12,8 @@ obj-y     = fork.o exec_domain.o exec_state.o panic.o \
 	    notifier.o ksysfs.o cred.o reboot.o \
 	    async.o range.o smpboot.o ucount.o regset.o ksyms_common.o
 
+obj-$(CONFIG_RECLAIMABLE_STACK) += stack_shrinker.o
+
 obj-$(CONFIG_MULTIUSER) += groups.o
 obj-$(CONFIG_VHOST_TASK) += vhost_task.o
 
kernel/fork.c +7 -0
index 6acad0038b78..9b2cc3d01dd1 100644
@@ -120,6 +120,8 @@
 /* For dup_mmap(). */
 #include "../mm/internal.h"
 
+#include "stack_shrinker.h"
+
 #include <trace/events/sched.h>
 
 #define CREATE_TRACE_POINTS
@@ -460,6 +462,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node)
 
 		tsk->stack_vm_area = vm_area;
 		tsk->stack = stack;
+		add_to_stack_shrinker(tsk, node);
 		return 0;
 	}
 
@@ -472,6 +475,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node)
 		free_vmap_stack(vm_area);
 		return -ENOMEM;
 	}
+
 	/*
 	 * We can't call find_vm_area() in interrupt context, and
 	 * free_thread_stack() can be called in interrupt context,
@@ -480,6 +484,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node)
 	tsk->stack_vm_area = vm_area;
 	stack = kasan_reset_tag(stack);
 	tsk->stack = stack;
+	add_to_stack_shrinker(tsk, node);
 	return 0;
 }
 
@@ -898,6 +903,7 @@ void __put_task_struct(struct task_struct *tsk)
 	delayacct_tsk_free(tsk);
 	put_signal_struct(tsk->signal);
 	sched_core_free(tsk);
+	remove_from_stack_shrinker(tsk);
 	free_task(tsk);
 }
 EXPORT_SYMBOL_GPL(__put_task_struct);
@@ -1124,6 +1130,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
 free_stack:
 	exit_task_stack_account(tsk);
 	free_thread_stack(tsk);
+	remove_from_stack_shrinker(tsk);
 free_tsk:
 	free_task_struct(tsk);
 	return NULL;
kernel/sched/core.c +15 -3
index 96226707c2f6..ab7db60d5dc2 100644
@@ -4252,6 +4252,7 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 {
 	guard(preempt)();
 	int cpu, success = 0;
+	bool need_deferred_repopulate, do_deferred_repopulate_wake = false;
 
 	wake_flags |= WF_TTWU;
 
@@ -4295,8 +4296,6 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 		if (!ttwu_state_match(p, state, &success))
 			break;
 
-		trace_sched_waking(p);
-
 		/*
 		 * Ensure we load p->on_rq _after_ p->state, otherwise it would
 		 * be possible to, falsely, observe p->on_rq == 0 and get stuck
@@ -4320,8 +4319,18 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 		 * A similar smp_rmb() lives in __task_needs_rq_lock().
 		 */
 		smp_rmb();
-		if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
+		if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) {
+			trace_sched_waking(p);
+			break;
+		}
+
+		if (!ensure_stack_is_present(p, &need_deferred_repopulate)) {
+			WRITE_ONCE(p->__state, TASK_STACK_RECLAIM);
+			do_deferred_repopulate_wake = need_deferred_repopulate;
 			break;
+		}
+
+		trace_sched_waking(p);
 
 		/*
 		 * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be
@@ -4418,6 +4427,8 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 	if (success)
 		ttwu_stat(p, task_cpu(p), wake_flags);
 
+	if (unlikely(do_deferred_repopulate_wake))
+		wake_stack_repopulate();
 	return success;
 }
 
@@ -5354,6 +5365,7 @@ static struct rq *finish_task_switch(struct task_struct *prev)
 	prev_state = READ_ONCE(prev->__state);
 	vtime_task_switch(prev);
 	perf_event_task_sched_in(prev, current);
+	allow_stack_reclaim(prev);
 	finish_task(prev);
 	tick_nohz_task_switch();
 	finish_lock_switch(rq);
kernel/sched/sched.h +3 -0
index 56acf502ba26..8a81fb83fe8f 100644
@@ -79,6 +79,7 @@
 #include <trace/events/sched.h>
 
 #include "../workqueue_internal.h"
+#include "../stack_shrinker.h"
 
 struct rq;
 struct cfs_rq;
@@ -3039,6 +3040,8 @@ static inline void __block_task(struct rq *rq, struct task_struct *p)
 		delayacct_blkio_start();
 	}
 
+	prepare_stack_for_reclaim(p);
+
 	ASSERT_EXCLUSIVE_WRITER(p->on_rq);
 
 	/*
kernel/stack_shrinker.c new file +464 -0

464 lines not on this page · raw message

kernel/stack_shrinker.h new file +58 -0

58 lines not on this page · raw message

Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#7 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Memory leak of `obj_cgroup` in the `copy_process()` error path. - [Medium] Repopulating kernel…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Memory leak of obj_cgroup in the copy_process() error path.
- [Medium] Repopulating kernel stack pages with NUMA_NO_NODE silently bypasses the task's NUMA mempolicy. --

commit 19da8486785bcd64133ab34a423835e83ba21942
Author: David Stevens <[email protected]>
Subject: Reclaim memory from blocked kernel stacks

This commit introduces a mechanism to reclaim unused portions of kernel stacks for blocked tasks. It adds scheduler hooks and a new task flag PF_RECLAIMABLE_STACK to manage safe stack reclaim and repopulation.

Quoted 4 lines
kernel/fork.c +0 -0
index 6acad0038b78e..9b2cc3d01dd10 100644

[ ... ]

kernel/fork.c static int alloc_thread_stack_node(struct task_struct *tsk, int node) · line 467

Quoted 7 lines
kernel/fork.c +1 -0
@@ -460,6 +462,7 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node)
 
 		tsk->stack_vm_area = vm_area;
 		tsk->stack = stack;
+		add_to_stack_shrinker(tsk, node);
 		return 0;
 	}

[Severity: High]
Does this leak the obj_cgroup reference in the copy_process() error path?

When dup_task_struct() completes successfully, we acquire an obj_cgroup reference inside add_to_stack_shrinker(). However, if a subsequent step in copy_process() fails (such as copy_mm()), the kernel jumps to the bad_fork_free label:

kernel/fork.c:copy_process() {
    ...
    bad_fork_free:
        WRITE_ONCE(p->__state, TASK_DEAD);
        exit_task_stack_account(p);
        put_task_stack(p);
        delayed_free_task(p);
    ...
}

Because delayed_free_task() directly frees the task without calling __put_task_struct() or remove_from_stack_shrinker(), it seems the obj_cgroup reference is permanently leaked, preventing the associated mem_cgroup from being destroyed.

Quoted 5 lines
kernel/stack_shrinker.c new file +0 -0
new file mode 100644
index 0000000000000..d7b1a7dfa7165

[ ... ]

Quoted 3 lines
kernel/stack_shrinker.c new file +3 -0
+static bool repopulate_stack(struct task_struct *tsk, bool is_deferred,
+			     struct llist_head *fail_list)
+{

[ ... ]

Quoted 2 lines
kernel/stack_shrinker.c new file +2 -0
+	if (is_deferred) {
+		gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO;

[ ... ]

kernel/stack_shrinker.c line 143

Quoted 2 lines
kernel/stack_shrinker.c new file +2 -0
+		for (; nr_allocated < num_missing_pages; nr_allocated++) {
+			pages[nr_allocated] = alloc_pages_node_noprof(node, gfp, 0);

[Severity: Medium]
Can this silently bypass the task's NUMA mempolicy?

If a task has no specific node forced, the target node is set to NUMA_NO_NODE.

Calling alloc_pages_node_noprof() with NUMA_NO_NODE unconditionally ignores the task's explicit NUMA mempolicy (like MPOL_BIND or MPOL_INTERLEAVE) and allocates from the current CPU's local node instead.

Should this branch and use alloc_pages_noprof() when the target node is NUMA_NO_NODE to properly respect the policy?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
maintainer
#8 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to David Stevens
Quoted 2 lines · David Stevens

A new TASK_STACK_RECLAIM state is introduced for tasks that are blocked waiting for the stacks to be repopulated.

Not yet read the full patch, but *why*.

maintainer
#9 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to David Stevens
Quoted 19 lines · David Stevens
kernel/sched/core.c +5 -3
@@ -4295,8 +4296,6 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 		if (!ttwu_state_match(p, state, &success))
 			break;
 
-		trace_sched_waking(p);
-
 		/*
 		 * Ensure we load p->on_rq _after_ p->state, otherwise it would
 		 * be possible to, falsely, observe p->on_rq == 0 and get stuck
@@ -4320,8 +4319,18 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 		 * A similar smp_rmb() lives in __task_needs_rq_lock().
 		 */
 		smp_rmb();
-		if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
+		if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) {
+			trace_sched_waking(p);
+			break;
+		}
+

kernel/sched/core.c int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) · line 4336

Quoted 10 lines
kernel/sched/core.c +6 -0
+		if (!ensure_stack_is_present(p, &need_deferred_repopulate)) {
+			WRITE_ONCE(p->__state, TASK_STACK_RECLAIM);
+			do_deferred_repopulate_wake = need_deferred_repopulate;
 			break;
+		}
+
+		trace_sched_waking(p);
 
 		/*
 		 * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be

Don't move that tracepoint. That's simply lying about how long it takes to wake up the task.

maintainer
#10 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to David Stevens
Quoted 7 lines · David Stevens
kernel/sched/core.c +2 -1
@@ -4320,8 +4319,18 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
 		 * A similar smp_rmb() lives in __task_needs_rq_lock().
 		 */
 		smp_rmb();
-		if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
+		if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) {
+			trace_sched_waking(p);

kernel/sched/core.c int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) · line 4333

Quoted 10 lines
kernel/sched/core.c +9 -0
+			break;
+		}
+
+		if (!ensure_stack_is_present(p, &need_deferred_repopulate)) {
+			WRITE_ONCE(p->__state, TASK_STACK_RECLAIM);
+			do_deferred_repopulate_wake = need_deferred_repopulate;
 			break;
+		}
+
+		trace_sched_waking(p);

Absolutely not; ensure_stack_is_present() must not call repopulate_stack() while holding ->pi_lock. Not happening.

#11 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · David Stevens
kernel/sched/core.c +7 -1
@@ -4320,8 +4319,18 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
               * A similar smp_rmb() lives in __task_needs_rq_lock().
               */
              smp_rmb();
-             if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
+             if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) {
+                     trace_sched_waking(p);
+                     break;
+             }
+
+             if (!ensure_stack_is_present(p, &need_deferred_repopulate)) {
+                     WRITE_ONCE(p->__state, TASK_STACK_RECLAIM);
Quoted 8 lines

kernel/sched/core.c int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) · line 4333

Quoted 5 lines
kernel/sched/core.c +4 -0
+                     do_deferred_repopulate_wake = need_deferred_repopulate;
                      break;
+             }
+
+             trace_sched_waking(p);

Absolutely not; ensure_stack_is_present() must not call repopulate_stack() while holding ->pi_lock. Not happening.

The optimistic fast path for repopulate_stack() could be modified to try pulling from a pre-allocated pool of zero'ed pages. That would reduce the function to a couple of memcg_kmem_charge_page() calls and then vmap_pages_range() to repopulate the stack's page tables. That wouldn't require touching any locks except a raw_spinlock protecting the pre-allocated pool (or just make it per_cpu). In terms of cost, this would involve a couple of atomic operations for the page pool lock and the memcg charging plus non-atomic operations on 5-10 other cache lines.

Is that within the scope of what can be done under the pi_lock? If that's still not happening, I can see how things look if we always defer wakeup to a workqueue.

-David

maintainer
#12 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 24 lines · David Stevens
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · David Stevens
kernel/sched/core.c +7 -1
@@ -4320,8 +4319,18 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
               * A similar smp_rmb() lives in __task_needs_rq_lock().
               */
              smp_rmb();
-             if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
+             if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) {
+                     trace_sched_waking(p);
+                     break;
+             }
+
+             if (!ensure_stack_is_present(p, &need_deferred_repopulate)) {
+                     WRITE_ONCE(p->__state, TASK_STACK_RECLAIM);
Quoted 8 lines

kernel/sched/core.c int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) · line 4333

Quoted 5 lines
kernel/sched/core.c +4 -0
+                     do_deferred_repopulate_wake = need_deferred_repopulate;
                      break;
+             }
+
+             trace_sched_waking(p);

Absolutely not; ensure_stack_is_present() must not call repopulate_stack() while holding ->pi_lock. Not happening.

Quoted 3 lines

The optimistic fast path for repopulate_stack() could be modified to try pulling from a pre-allocated pool of zero'ed pages. That would reduce the function to a couple of memcg_kmem_charge_page() calls and

Afaict memcg_kmem_charge_page() ends up in a local_lock, which is a spinlock, so that cannot be.

Most, if not everything, in mm/ is build around being preemptible and thus not suitable for use under raw_spinlock_t.

Quoted 1 line

then vmap_pages_range() to repopulate the stack's page tables. That

vmap_page_range() can end up in the allocator, which I suppose is ruled out by the vmap having been populated before, but it still has a might_sleep() that will scream AFAICT.

Quoted 9 lines

wouldn't require touching any locks except a raw_spinlock protecting the pre-allocated pool (or just make it per_cpu). In terms of cost, this would involve a couple of atomic operations for the page pool lock and the memcg charging plus non-atomic operations on 5-10 other cache lines.

Is that within the scope of what can be done under the pi_lock? If that's still not happening, I can see how things look if we always defer wakeup to a workqueue.

As long as it really is all atomics it should be fine. If there is a lock, it must be raw_spinlock_t, but ideally no new locks nested under pi_lock.

maintainer
#13 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 6 lines · Peter Zijlstra
Quoted 3 lines

The optimistic fast path for repopulate_stack() could be modified to try pulling from a pre-allocated pool of zero'ed pages. That would reduce the function to a couple of memcg_kmem_charge_page() calls and

Afaict memcg_kmem_charge_page() ends up in a local_lock, which is a spinlock, so that cannot be.

Perhaps just don't uncharge them on reclaim, then you don't have to re-charge.

#14 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to Peter Zijlstra
Quoted 30 lines · Peter Zijlstra
Quoted 24 lines · David Stevens
Quoted 14 lines · Peter Zijlstra
Quoted 12 lines · David Stevens
kernel/sched/core.c +7 -1
@@ -4320,8 +4319,18 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
               * A similar smp_rmb() lives in __task_needs_rq_lock().
               */
              smp_rmb();
-             if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags))
+             if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) {
+                     trace_sched_waking(p);
+                     break;
+             }
+
+             if (!ensure_stack_is_present(p, &need_deferred_repopulate)) {
+                     WRITE_ONCE(p->__state, TASK_STACK_RECLAIM);
Quoted 8 lines

kernel/sched/core.c int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) · line 4333

Quoted 5 lines
kernel/sched/core.c +4 -0
+                     do_deferred_repopulate_wake = need_deferred_repopulate;
                      break;
+             }
+
+             trace_sched_waking(p);

Absolutely not; ensure_stack_is_present() must not call repopulate_stack() while holding ->pi_lock. Not happening.

Quoted 3 lines

The optimistic fast path for repopulate_stack() could be modified to try pulling from a pre-allocated pool of zero'ed pages. That would reduce the function to a couple of memcg_kmem_charge_page() calls and

Quoted 5 lines

Afaict memcg_kmem_charge_page() ends up in a local_lock, which is a spinlock, so that cannot be.

Most, if not everything, in mm/ is build around being preemptible and thus not suitable for use under raw_spinlock_t.

With the recent work to make memcg charging NMI safe, I had thought that charging from within the scheduler would be safe. However, you're right that the fast path does call local_trylock(). I didn't realize that only works because of an in_nmi() check inside local_trylock().

Given that NMI safe charging requires only using trylock and handling the !trylock case, it's not infeasible to have an implementation that always skips the trylocks.

The alternatives to that would be either not uncharging on unmap, as suggested further down the thread, or using deferred charging. However, both of those approaches would make memcg charging less accurate, which is not desirable.

Quoted 5 lines
Quoted 1 line

then vmap_pages_range() to repopulate the stack's page tables. That

vmap_page_range() can end up in the allocator, which I suppose is ruled out by the vmap having been populated before, but it still has a might_sleep() that will scream AFAICT.

It looks like when we go through vmap_small_pages_range_noflush(), the might_sleep() check is missing. That would explain why I haven't been seeing errors.

We wouldn't want to remove that might_sleep(), so I guess this would require either a new function or flag to support mapping into existing page tables.

-David

maintainer
#15 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to David Stevens
Quoted 4 lines · David Stevens

This feature does work on PREEMPT_RT, but is likely undesirable due to the extra uncertainty. The fact that alloc_pages_nolock_noprof() cannot be called from under the scheduler lock also makes repopulating stacks more expensive.

Aaah, so you knew it was hot garbage :-(

Please don't ever post anything like this again.

maintainer
#16 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to David Stevens
Quoted 38 lines · David Stevens
(quoted diff) +36 -0
+DEFINE_CLASS(allow_stack_reclaim, bool,
+	     ({
+		if (!_T)
+			current->flags &= ~PF_RECLAIMABLE_STACK;
+	      }),
+	     ({
+		bool was_set = current->flags & PF_RECLAIMABLE_STACK;
+
+		current->flags |= PF_RECLAIMABLE_STACK;
+		was_set;
+	      }),
+	     void)
 
+	allow_stack_reclaim(prev);
 
+void __allow_stack_reclaim(struct task_struct *tsk)
+{
+	union stack_reclaim_state prev_state, target_state;
+
+	if (WARN_ON_ONCE(tsk->__state == TASK_DEAD))
+		return;
+
+	prev_state.val = READ_ONCE(tsk->stack_reclaim_state.val);
+	do {
+		target_state.val = prev_state.val;
+
+		if (prev_state.stack_state != STACK_PREPARE_RECLAIM) {
+			WARN(prev_state.stack_state != STACK_IN_USE,
+			     "Reclaimable state %x for previously running task", prev_state.val);
+			return;
+		}
+		target_state.stack_state = STACK_RECLAIMABLE;
+	} while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, target_state.val));
+
+	if (irq_work_queue(&tsk->stack_reclaim_work->irq_work)) {
+		/*
+		 * Take a ref that gets released by do_reclaim_stack() so we don't
+		 * have to worry about races with remove_from_stack_shrinker().

Review

Quoted 10 lines
(quoted diff) +9 -0
+		 */
+		get_task_struct(tsk);
+	}
+}
 
+static inline void allow_stack_reclaim(struct task_struct *tsk)
+{
+	if (unlikely(tsk->flags & PF_RECLAIMABLE_STACK))
+		__allow_stack_reclaim(tsk);
+}

So you have a guard with the same name as a function, but the function only functions when inside the guard of the same name. WTF ?!

Anyway, it looks like you're sprinkling this guard around a few specific block sites. Which seems to suggest your PF_ flag *should* have been a TASK_ flag, no?

#17 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 54 lines · Peter Zijlstra
Quoted 38 lines · David Stevens
(quoted diff) +36 -0
+DEFINE_CLASS(allow_stack_reclaim, bool,
+          ({
+             if (!_T)
+                     current->flags &= ~PF_RECLAIMABLE_STACK;
+           }),
+          ({
+             bool was_set = current->flags & PF_RECLAIMABLE_STACK;
+
+             current->flags |= PF_RECLAIMABLE_STACK;
+             was_set;
+           }),
+          void)
 
+     allow_stack_reclaim(prev);
 
+void __allow_stack_reclaim(struct task_struct *tsk)
+{
+     union stack_reclaim_state prev_state, target_state;
+
+     if (WARN_ON_ONCE(tsk->__state == TASK_DEAD))
+             return;
+
+     prev_state.val = READ_ONCE(tsk->stack_reclaim_state.val);
+     do {
+             target_state.val = prev_state.val;
+
+             if (prev_state.stack_state != STACK_PREPARE_RECLAIM) {
+                     WARN(prev_state.stack_state != STACK_IN_USE,
+                          "Reclaimable state %x for previously running task", prev_state.val);
+                     return;
+             }
+             target_state.stack_state = STACK_RECLAIMABLE;
+     } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, target_state.val));
+
+     if (irq_work_queue(&tsk->stack_reclaim_work->irq_work)) {
+             /*
+              * Take a ref that gets released by do_reclaim_stack() so we don't
+              * have to worry about races with remove_from_stack_shrinker().

Review

Quoted 10 lines
(quoted diff) +9 -0
+              */
+             get_task_struct(tsk);
+     }
+}
 
+static inline void allow_stack_reclaim(struct task_struct *tsk)
+{
+     if (unlikely(tsk->flags & PF_RECLAIMABLE_STACK))
+             __allow_stack_reclaim(tsk);
+}

So you have a guard with the same name as a function, but the function only functions when inside the guard of the same name. WTF ?!

Quoted 3 lines

Anyway, it looks like you're sprinkling this guard around a few specific block sites. Which seems to suggest your PF_ flag *should* have been a TASK_ flag, no?

A TASK_ flag is definitely better. I was thinking in terms of defining safe scopes, but that just complicates things for no real benefit.

-David

#18 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to David Stevens

arch/Kconfig config VMAP_STACK · line 1541

Quoted 13 lines · David Stevens
arch/Kconfig +5 -0
index fa7507ac8e13..adb4a5957996 100644
@@ -1534,6 +1534,24 @@ config VMAP_STACK
 	  backing virtual mappings with real shadow memory, and KASAN_VMALLOC
 	  must be enabled.
 
+config HAVE_ARCH_RECLAIMABLE_STACK
+	def_bool n
+
+config RECLAIMABLE_STACK
+	default !PREEMPT_RT && !PROC_KCORE

This shouldn't default like this for RT. It either is useable or it is not.

arch/Kconfig config VMAP_STACK · line 1553

Quoted 12 lines
arch/Kconfig +12 -0
+	bool "Allow stacks of some blocked threads to be reclaimed"
+	depends on VMAP_STACK && !STACK_GROWSUP
+	depends on HAVE_ARCH_RECLAIMABLE_STACK
+	depends on !DEBUG_STACK_USAGE
+	depends on !KASAN_VMALLOC # TODO: add support for this
+	depends on !DEBUG_KMEMLEAK # TODO: add support for this
+	help
+	  Enable this to allow the unused portion of kernel stacks of most
+	  blocked tasks to be reclaimed.
+
+	  The wakeup latency of tasks with reclaimed stacks may increase,
+	  especially while the system is under memory pressure.

It says *may* increase and on RT it _definitely_ will increase since there is a kworker involved not to mention the memory allocation itself. Anyway. This either needs to stay away from PREEMPT_RT or find a way to exclude at the very least mlock()ed tasks.
Did lockdep see this?

If I understood the whole exercise correct then you have a kernel stack of two pages and in best case you can unmap and release the second page while the task is napping.

What might be a tad simpler is to memset(,0,) the remaining part of the stack. Since the stack is vmap-ed it should be swapped out on its own without additional tricks. That memset() would help zram to compress better so it uses less memory. ta-da.

What also should be simpler (and I am not saying just to move you away from the scheduler) is to have a shrinker which iterates over all tasks which are marked for reclaim and then similar to swap just unmap both stack pages and release the second page which is not used. Upon wake up the task should create a page_fault which would be used to allocate the second stack page and map the whole stack again.

This sounds simpler.

Sebastian

maintainer
#19 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 28 lines · Sebastian Andrzej Siewior

arch/Kconfig config VMAP_STACK · line 1541

Quoted 13 lines · David Stevens
arch/Kconfig +5 -0
index fa7507ac8e13..adb4a5957996 100644
@@ -1534,6 +1534,24 @@ config VMAP_STACK
 	  backing virtual mappings with real shadow memory, and KASAN_VMALLOC
 	  must be enabled.
 
+config HAVE_ARCH_RECLAIMABLE_STACK
+	def_bool n
+
+config RECLAIMABLE_STACK
+	default !PREEMPT_RT && !PROC_KCORE

This shouldn't default like this for RT. It either is useable or it is not.

Quoted 10 lines
arch/Kconfig +10 -0
+	bool "Allow stacks of some blocked threads to be reclaimed"
+	depends on VMAP_STACK && !STACK_GROWSUP
+	depends on HAVE_ARCH_RECLAIMABLE_STACK
+	depends on !DEBUG_STACK_USAGE
+	depends on !KASAN_VMALLOC # TODO: add support for this
+	depends on !DEBUG_KMEMLEAK # TODO: add support for this
+	help
+	  Enable this to allow the unused portion of kernel stacks of most
+	  blocked tasks to be reclaimed.
+
Quoted 8 lines

arch/Kconfig config VMAP_STACK · line 1553

Quoted 2 lines
arch/Kconfig +2 -0
+	  The wakeup latency of tasks with reclaimed stacks may increase,
+	  especially while the system is under memory pressure.

It says *may* increase and on RT it _definitely_ will increase since there is a kworker involved not to mention the memory allocation itself. Anyway. This either needs to stay away from PREEMPT_RT or find a way to exclude at the very least mlock()ed tasks.
Did lockdep see this?

It should have. They're taking spinlock inside raw_spinlock and lockdep should very much warn about that by default.

Quoted 3 lines

If I understood the whole exercise correct then you have a kernel stack of two pages and in best case you can unmap and release the second page while the task is napping.

THREAD_SIZE_ORDER 2
THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)

that makes for 4 pages.

Quoted 4 lines

What might be a tad simpler is to memset(,0,) the remaining part of the stack. Since the stack is vmap-ed it should be swapped out on its own without additional tricks. That memset() would help zram to compress better so it uses less memory. ta-da.

That would still be a 12k memset with IRQs-disabled and rq->lock held.

Quoted 6 lines

What also should be simpler (and I am not saying just to move you away from the scheduler) is to have a shrinker which iterates over all tasks which are marked for reclaim and then similar to swap just unmap both stack pages and release the second page which is not used. Upon wake up the task should create a page_fault which would be used to allocate the second stack page and map the whole stack again.

Right, so you can FREEZE the task, unmap its stack and then thaw it or something. But there should be a definite opt-out on all this, because taking faults on your stack will be horrible.

Not to mention you'll suffer wakeup latencies while frozen.

This all really sounds like what should be addressed is this insane number of tasks rather than trying to cope with the consequences of that.

maintainer
#20 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 11 lines · Peter Zijlstra
Quoted 6 lines

What also should be simpler (and I am not saying just to move you away from the scheduler) is to have a shrinker which iterates over all tasks which are marked for reclaim and then similar to swap just unmap both stack pages and release the second page which is not used. Upon wake up the task should create a page_fault which would be used to allocate the second stack page and map the whole stack again.

Right, so you can FREEZE the task, unmap its stack and then thaw it or something. But there should be a definite opt-out on all this, because taking faults on your stack will be horrible.

Quoted 5 lines

Not to mention you'll suffer wakeup latencies while frozen.

This all really sounds like what should be addressed is this insane number of tasks rather than trying to cope with the consequences of that.

Taking faults on the task-stack is not going to work. That fault will happen while you have locks held and IRQs disabled. Ideally, it'll happen when you're knee deep in the allocator.

That fault will then again call the allocator to allocator your stack page, and...

#21 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 6 lines · Peter Zijlstra

Taking faults on the task-stack is not going to work. That fault will happen while you have locks held and IRQs disabled. Ideally, it'll happen when you're knee deep in the allocator.

That fault will then again call the allocator to allocator your stack page, and...

indeed. So I do have a few ideas floating around but I leave it at it.

Sebastian

#22 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines

arch/Kconfig config VMAP_STACK · line 1553

Quoted 3 lines
arch/Kconfig +3 -0
+
+	  The wakeup latency of tasks with reclaimed stacks may increase,
+	  especially while the system is under memory pressure.

It says *may* increase and on RT it _definitely_ will increase since there is a kworker involved not to mention the memory allocation itself. Anyway. This either needs to stay away from PREEMPT_RT or find a way to exclude at the very least mlock()ed tasks.
Did lockdep see this?

Quoted 2 lines

It should have. They're taking spinlock inside raw_spinlock and lockdep should very much warn about that by default.

Yes. My point was that this was hidden from lockdep.

Quoted 8 lines
Quoted 3 lines

If I understood the whole exercise correct then you have a kernel stack of two pages and in best case you can unmap and release the second page while the task is napping.

THREAD_SIZE_ORDER 2
THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)

that makes for 4 pages.

Oh. I wasn't aware that we have 16kib stacks these days. But looking at it we have it now for over 10 years… Judging from 6538b8ea886e4 ("x86_64: expand kernel stack to 16K") it might be temporary and things are better now? Arm64 has a different story according to 845ad05ec31e0 ("arm64: Change kernel stack size to 16K"). Risc-V also mentions "for now" in 0cac21b02ba5f ("riscv: use 16KB kernel stack on 64-bit").

I just booted my XFS kvm box and did things and 8KiB works so far.

Quoted 6 lines
Quoted 4 lines

What might be a tad simpler is to memset(,0,) the remaining part of the stack. Since the stack is vmap-ed it should be swapped out on its own without additional tricks. That memset() would help zram to compress better so it uses less memory. ta-da.

That would still be a 12k memset with IRQs-disabled and rq->lock held.

Right, because the stack grew a bit. Probably still cheaper compared to the other things done here ;)

Quoted 10 lines
Quoted 6 lines

What also should be simpler (and I am not saying just to move you away from the scheduler) is to have a shrinker which iterates over all tasks which are marked for reclaim and then similar to swap just unmap both stack pages and release the second page which is not used. Upon wake up the task should create a page_fault which would be used to allocate the second stack page and map the whole stack again.

Right, so you can FREEZE the task, unmap its stack and then thaw it or something. But there should be a definite opt-out on all this, because taking faults on your stack will be horrible.

Definitely. Not something for the currently visible app.

Quoted 1 line

Not to mention you'll suffer wakeup latencies while frozen.

Right but you would use it under memory pressure and steal the stack from the most idle tasks rather from everyone.

Quoted 3 lines

This all really sounds like what should be addressed is this insane number of tasks rather than trying to cope with the consequences of that.

Sebastian

maintainer
#23 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 5 lines · On Fri, 28 Aug 2026 17:10:18 +0200 Sebastian Andrzej Siewior…
Quoted 4 lines

THREAD_SIZE_ORDER 2
THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)

that makes for 4 pages.

Quoted 8 lines

Oh. I wasn't aware that we have 16kib stacks these days. But looking at it we have it now for over 10 years… Judging from 6538b8ea886e4 ("x86_64: expand kernel stack to 16K") it might be temporary and things are better now? Arm64 has a different story according to 845ad05ec31e0 ("arm64: Change kernel stack size to 16K"). Risc-V also mentions "for now" in 0cac21b02ba5f ("riscv: use 16KB kernel stack on 64-bit").

I just booted my XFS kvm box and did things and 8KiB works so far.

If you want to see how big your stack is and where its used, you can run the stack tracer:

 # trace-cmd stack --start

[wait]

 # trace-cmd stack
(stack tracer running)
        Depth    Size   Location    (41 entries)
        -----    ----   --------
  0)     5032      48   choose_idle_cpu+0x5/0x100
  1)     4984     296   select_task_rq_fair+0xa84/0x2b40
  2)     4688     104   try_to_wake_up+0x168/0x7e0
  3)     4584      16   task_work_add+0xd7/0xf0
  4)     4568      16   io_req_normal_work_add+0x75/0xb0
  5)     4552      40   io_poll_wake+0x10f/0x160
  6)     4512      64   __wake_up_common+0x72/0xa0
  7)     4448      40   __wake_up_sync_key+0x43/0x60
  8)     4408      24   sock_def_readable+0x46/0xe0
  9)     4384      80   tun_net_xmit+0x240/0x550 [tun]
 10)     4304      80   dev_hard_start_xmit+0x63/0x1e0
 11)     4224     256   __dev_queue_xmit+0x880/0x10b0
 12)     3968      24   br_dev_queue_push_xmit+0x62/0xf0 [bridge]
 13)     3944      80   br_dev_xmit+0x15e/0x4a0 [bridge]
 14)     3864      80   dev_hard_start_xmit+0x63/0x1e0
 15)     3784     256   __dev_queue_xmit+0x880/0x10b0
 16)     3528      72   ip_finish_output2+0x2ce/0x650
 17)     3456     104   ip_output+0x63/0x110
 18)     3352      72   __ip_queue_xmit+0x16f/0x470
 19)     3280     232   __tcp_transmit_skb+0xcb5/0x1140
 20)     3048     120   tcp_write_xmit+0x5aa/0x1760
 21)     2928      24   __tcp_push_pending_frames+0x39/0x110
 22)     2904     104   tcp_rcv_established+0x37f/0xeb0
 23)     2800      32   tcp_v4_do_rcv+0x13f/0x300
 24)     2768     168   tcp_v4_rcv+0xc01/0x1300
 25)     2600      48   ip_protocol_deliver_rcu+0x35/0x1b0
 26)     2552      24   ip_local_deliver_finish+0x85/0x100
 27)     2528      40   __netif_receive_skb_one_core+0x85/0xa0
 28)     2488      56   netif_receive_skb+0x127/0x180
 29)     2432     104   br_handle_frame_finish+0x428/0x680 [bridge]
 30)     2328      40   br_handle_frame+0x132/0x2a0 [bridge]
 31)     2288     280   __netif_receive_skb_core.constprop.0+0x16e/0xdf0
 32)     2008      40   __netif_receive_skb_one_core+0x39/0xa0
 33)     1968      56   netif_receive_skb+0x127/0x180
 34)     1912     256   tun_get_user+0xbd0/0x1260 [tun]
 35)     1656      56   tun_chr_write_iter+0x77/0xba [tun]
 36)     1600      88   do_iter_readv_writev+0x161/0x260
 37)     1512     256   vfs_writev+0x168/0x3c0
 38)     1256      80   do_writev+0x7f/0x110
 39)     1176     984   do_syscall_64+0xaa/0x670
 40)      192     192   entry_SYSCALL_64_after_hwframe+0x76/0x7e

Note, it only monitors task context (not interrupt).

-- Steve

maintainer
#24 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 51 lines · On Fri, 28 Aug 2026 15:08:36 -0400 Steven Rostedt <[email protected]> wrote:

If you want to see how big your stack is and where its used, you can run the stack tracer:

 # trace-cmd stack --start

[wait]

 # trace-cmd stack
(stack tracer running)
        Depth    Size   Location    (41 entries)
        -----    ----   --------
  0)     5032      48   choose_idle_cpu+0x5/0x100
  1)     4984     296   select_task_rq_fair+0xa84/0x2b40
  2)     4688     104   try_to_wake_up+0x168/0x7e0
  3)     4584      16   task_work_add+0xd7/0xf0
  4)     4568      16   io_req_normal_work_add+0x75/0xb0
  5)     4552      40   io_poll_wake+0x10f/0x160
  6)     4512      64   __wake_up_common+0x72/0xa0
  7)     4448      40   __wake_up_sync_key+0x43/0x60
  8)     4408      24   sock_def_readable+0x46/0xe0
  9)     4384      80   tun_net_xmit+0x240/0x550 [tun]
 10)     4304      80   dev_hard_start_xmit+0x63/0x1e0
 11)     4224     256   __dev_queue_xmit+0x880/0x10b0
 12)     3968      24   br_dev_queue_push_xmit+0x62/0xf0 [bridge]
 13)     3944      80   br_dev_xmit+0x15e/0x4a0 [bridge]
 14)     3864      80   dev_hard_start_xmit+0x63/0x1e0
 15)     3784     256   __dev_queue_xmit+0x880/0x10b0
 16)     3528      72   ip_finish_output2+0x2ce/0x650
 17)     3456     104   ip_output+0x63/0x110
 18)     3352      72   __ip_queue_xmit+0x16f/0x470
 19)     3280     232   __tcp_transmit_skb+0xcb5/0x1140
 20)     3048     120   tcp_write_xmit+0x5aa/0x1760
 21)     2928      24   __tcp_push_pending_frames+0x39/0x110
 22)     2904     104   tcp_rcv_established+0x37f/0xeb0
 23)     2800      32   tcp_v4_do_rcv+0x13f/0x300
 24)     2768     168   tcp_v4_rcv+0xc01/0x1300
 25)     2600      48   ip_protocol_deliver_rcu+0x35/0x1b0
 26)     2552      24   ip_local_deliver_finish+0x85/0x100
 27)     2528      40   __netif_receive_skb_one_core+0x85/0xa0
 28)     2488      56   netif_receive_skb+0x127/0x180
 29)     2432     104   br_handle_frame_finish+0x428/0x680 [bridge]
 30)     2328      40   br_handle_frame+0x132/0x2a0 [bridge]
 31)     2288     280   __netif_receive_skb_core.constprop.0+0x16e/0xdf0
 32)     2008      40   __netif_receive_skb_one_core+0x39/0xa0
 33)     1968      56   netif_receive_skb+0x127/0x180
 34)     1912     256   tun_get_user+0xbd0/0x1260 [tun]
 35)     1656      56   tun_chr_write_iter+0x77/0xba [tun]
 36)     1600      88   do_iter_readv_writev+0x161/0x260
 37)     1512     256   vfs_writev+0x168/0x3c0
 38)     1256      80   do_writev+0x7f/0x110
Quoted 1 line

39) 1176 984 do_syscall_64+0xaa/0x670

If one worries about stack size, they may want to turn off CONFIG_RANDOMIZE_KSTACK_OFFSET as I see some big numbers for do_syscall_64()

-- Steve

Quoted 1 line

40) 192 192 entry_SYSCALL_64_after_hwframe+0x76/0x7e

maintainer
#25 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 2 lines · On Fri, 28 Aug 2026 15:13:13 -0400 Steven Rostedt <[email protected]> wrote:

If one worries about stack size, they may want to turn off CONFIG_RANDOMIZE_KSTACK_OFFSET as I see some big numbers for do_syscall_64()

And the scheduler has some heavy stack usage here!

  # trace-cmd stack
(stack tracer running)
        Depth    Size   Location    (31 entries)
        -----    ----   --------
  0)     8120      48   __msecs_to_jiffies+0x9/0x30
  1)     8072     104   update_group_capacity+0x94/0x960
  2)     7968     528   update_sd_lb_stats.constprop.0+0x426/0x39b0
  3)     7440     424   sched_balance_find_src_group+0x8f/0x1150
  4)     7016     552   sched_balance_rq+0x934/0x4130
  5)     6464     408   pick_task_fair+0xa71/0x1fe0
  6)     6056     496   __schedule+0x628/0x76f0
  7)     5560      32   schedule+0xde/0x2c0
  8)     5528      32   io_schedule+0x8c/0x100
  9)     5496     256   rq_qos_wait+0x12b/0x230
 10)     5240     136   wbt_wait+0x150/0x260
 11)     5104      40   __rq_qos_throttle+0x51/0xa0
 12)     5064     304   blk_mq_submit_bio+0xc40/0x28e0
 13)     4760     240   submit_bio_noacct_nocheck+0x410/0xb30
 14)     4520      40   ext4_io_submit+0xca/0x1a0
 15)     4480     144   ext4_bio_write_folio+0x5e2/0x1470
 16)     4336      96   mpage_process_page_bufs+0x392/0x700
 17)     4240     616   mpage_prepare_extent_to_map+0xaae/0x1120
 18)     3624     480   ext4_do_writepages+0x971/0x37f0
 19)     3144     312   ext4_writepages+0x2d2/0x610
 20)     2832     152   do_writepages+0x21e/0x560
 21)     2680     184   __writeback_single_inode+0x119/0x1240
 22)     2496     752   writeback_sb_inodes+0x67d/0x17b0
 23)     1744     168   __writeback_inodes_wb+0xf2/0x270
 24)     1576     304   wb_writeback+0x63e/0x890
 25)     1272     320   wb_workfn+0x844/0xca0
 26)      952     360   process_one_work+0x8b4/0x15d0
 27)      592     176   worker_thread+0x5d3/0xfb0
 28)      416      64   kthread+0x33c/0x420
 29)      352     160   ret_from_fork+0x65c/0x9e0
 30)      192     192   ret_from_fork_asm+0x1a/0x30

Note, I'm running this on my server. That is, this is a production machine.

-- Steve

#26 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 6 lines · Steven Rostedt
Quoted 2 lines · On Fri, 28 Aug 2026 15:13:13 -0400 Steven Rostedt <[email protected]> wrote:

If one worries about stack size, they may want to turn off CONFIG_RANDOMIZE_KSTACK_OFFSET as I see some big numbers for do_syscall_64()

Quoted 7 lines

And the scheduler has some heavy stack usage here!

  # trace-cmd stack
(stack tracer running)
        Depth    Size   Location    (31 entries)
        -----    ----   --------
  0)     8120      48   __msecs_to_jiffies+0x9/0x30

I didn't expect get_user_pages_unlocked() getting that big:

(stack tracer running)

        Depth    Size   Location    (60 entries)
        -----    ----   --------
  0)     8176      48   __css_rstat_updated+0x9/0x150
  1)     8128       8   __cgroup_account_cputime+0x2f/0x50
  2)     8120      48   update_se+0x10c/0x1b0
  3)     8072      48   update_curr+0x31/0x180
  4)     8024      56   put_prev_entity+0x153/0x1f0
  5)     7968     136   pick_next_task_fair+0x684/0x8e0
  6)     7832     120   __schedule+0x1ca/0x1130
  7)     7712       8   preempt_schedule_irq+0x38/0x60
  8)     7704      72   irqentry_exit+0x148/0x6b0
  9)     7632     136   asm_sysvec_apic_timer_interrupt+0x1a/0x20
 10)     7496     240   HUF_compress1X_usingCTable_internal_bmi2+0x12d2/0x1c40
 11)     7256      80   HUF_compress4X_usingCTable_internal+0x192/0x1d0
 12)     7176      32   HUF_compressCTable_internal+0x61/0x90
 13)     7144     136   HUF_compress_internal+0x2d2/0x500
 14)     7008      56   HUF_compress4X_repeat+0x25/0x60
 15)     6952     176   ZSTD_compressLiterals+0x1ad/0x400
 16)     6776     224   ZSTD_entropyCompressSeqStore_internal+0xfb/0x320
 17)     6552     112   ZSTD_compressBlock_internal+0x11b/0x210
 18)     6440     296   ZSTD_compressContinue_internal+0x232/0xdd0
 19)     6144      56   ZSTD_compressEnd_public+0x27/0x190
 20)     6088     112   ZSTD_compressStream2+0x9fc/0xa60
 21)     5976      80   ZSTD_compress2+0x83/0xd0
 22)     5896      24   zstd_compress+0x3d/0x70 [zram]
 23)     5872      72   zcomp_compress+0x62/0x90 [zram]
 24)     5800     144   zram_submit_bio+0x558/0xb00 [zram]
 25)     5656     128   __submit_bio+0x164/0x240
 26)     5528      96   submit_bio_noacct_nocheck+0x134/0x390
 27)     5432      64   bio_await+0xa7/0xb0
 28)     5368      16   submit_bio_wait+0x16/0x20
 29)     5352     184   swap_writepage_bdev_sync.isra.0+0x12e/0x230
 30)     5168      40   swap_writeout+0x18d/0x2e0
 31)     5128     176   shmem_writeout+0x286/0x530
 32)     4952     440   shrink_folio_list+0x588/0xfc0
 33)     4512     296   evict_folios+0x399/0xac0
 34)     4216     160   try_to_shrink_lruvec+0x1a4/0x390
 35)     4056      64   shrink_one+0xc0/0x1a0
 36)     3992     240   shrink_node+0xa9a/0xcf0
 37)     3752      88   do_try_to_free_pages+0xb3/0x4d0
 38)     3664     176   try_to_free_pages+0xce/0x220
 39)     3488     264   __alloc_pages_slowpath.constprop.0+0x8cc/0x1240
 40)     3224      96   __alloc_frozen_pages_noprof+0x2f6/0x340
 41)     3128      64   alloc_pages_mpol+0xb6/0x190
 42)     3064      56   vma_alloc_folio_noprof+0x6e/0xd0
 43)     3008      80   do_anonymous_page+0x32e/0x8f0
 44)     2928     224   __handle_mm_fault+0xb31/0xf60
 45)     2704      64   handle_mm_fault+0xee/0x2f0
 46)     2640     152   __get_user_pages+0x1b5/0x1140
 47)     2488     104   get_user_pages_unlocked+0xf0/0x330
 48)     2384     128   hva_to_pfn+0x2ea/0x450 [kvm]
 49)     2256      64   __kvm_faultin_pfn+0x61/0x90 [kvm]
 50)     2192     120   kvm_mmu_faultin_pfn+0x2cf/0x6f0 [kvm]
 51)     2072      24   kvm_tdp_page_fault+0x94/0xf0 [kvm]
 52)     2048     120   kvm_mmu_do_page_fault+0x1d9/0x210 [kvm]
 53)     1928     200   kvm_mmu_page_fault+0x7e/0x7b0 [kvm]
 54)     1728      40   npf_interception+0xba/0x240 [kvm_amd]
 55)     1688     168   kvm_arch_vcpu_ioctl_run+0x931/0x1900 [kvm]
 56)     1520     200   kvm_vcpu_ioctl+0x2e4/0xa10 [kvm]
 57)     1320      56   __x64_sys_ioctl+0x97/0xe0
 58)     1264    1072   do_syscall_64+0xe1/0x640
 59)      192     192   entry_SYSCALL_64_after_hwframe+0x76/0x7e

Does this stack tracer distinguish between the kernel-stack and IRQ-stack? I would expect asm_sysvec_apic_timer_interrupt() on the IRQ-stack and not adding weight to the kernel stack. Given how close this is to 8192, I don't think two stack pages will happen soon.

Sebastian

maintainer
#27 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 16 lines · Sebastian Andrzej Siewior

(stack tracer running)

        Depth    Size   Location    (60 entries)
        -----    ----   --------
  0)     8176      48   __css_rstat_updated+0x9/0x150
  1)     8128       8   __cgroup_account_cputime+0x2f/0x50
  2)     8120      48   update_se+0x10c/0x1b0
  3)     8072      48   update_curr+0x31/0x180
  4)     8024      56   put_prev_entity+0x153/0x1f0
  5)     7968     136   pick_next_task_fair+0x684/0x8e0
  6)     7832     120   __schedule+0x1ca/0x1130
  7)     7712       8   preempt_schedule_irq+0x38/0x60
  8)     7704      72   irqentry_exit+0x148/0x6b0
  9)     7632     136   asm_sysvec_apic_timer_interrupt+0x1a/0x20
 10)     7496     240   HUF_compress1X_usingCTable_internal_bmi2+0x12d2/0x1c40
Quoted 3 lines

Does this stack tracer distinguish between the kernel-stack and IRQ-stack? I would expect asm_sysvec_apic_timer_interrupt() on the IRQ-stack and not adding weight to the kernel stack.

This is the return from interrupt path, irqentry_exit() is right after we switch back from the IRQ stack and are back on the task stack. We have to be, in order to schedule the task and all that.

maintainer
#28 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to Sebastian Andrzej Siewior
Quoted 2 lines · On Mon, 31 Aug 2026 08:55:09 +0200 Sebastian Andrzej Siewior…

Does this stack tracer distinguish between the kernel-stack and IRQ-stack?

It should only look at kernel-stack and not the irq stack. This stack trace may have triggered when the interrupt put back the kernel stack, as it is in the irqentry_exit code and about to do a schedule. The code has:

kernel/trace/trace_stack.c:

	/* we do not handle interrupt stacks yet */
	if (!object_is_on_stack(stack))
		return;

Which returns if the object is on the kernel-stack, and here "stack" is the parameter passed into this function.

Quoted 2 lines

I would expect asm_sysvec_apic_timer_interrupt() on the IRQ-stack and not adding weight to the kernel stack.

Right. But I believe this is where it switched back to the kernel-stack.

Quoted 2 lines

Given how close this is to 8192, I don't think two stack pages will happen soon.

Agreed :-(

-- Steve

maintainer
#29 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to Steven Rostedt
Quoted 4 lines · Steven Rostedt
  1)     8072     104   update_group_capacity+0x94/0x960
  2)     7968     528   update_sd_lb_stats.constprop.0+0x426/0x39b0
  3)     7440     424   sched_balance_find_src_group+0x8f/0x1150
  4)     7016     552   sched_balance_rq+0x934/0x4130

Bah, yeah, those on-stack statistics just keep growing.

This should probably help. Very lightly tested. Also we can probably relax the assertion to bh-disabled and avoid the extra irq-disable around sched_balance_rq().

Anybody got time to play around with this?

---

kernel/sched/fair.c +67 -33
index 8dff37059faf..0c83b0856a95 100644
@@ -11418,6 +11418,39 @@ struct sd_lb_stats {
 	struct sg_lb_stats local_stat;		/* Statistics of the local group */
 };
 
+struct pcpu_lb_stats {
+	struct sd_lb_stats sds;
+	struct sg_lb_stats sgs;
+	struct sg_lb_stats local_sgs;
+	struct sg_lb_stats idlest_sgs;
+};
+
+static DEFINE_PER_CPU(struct pcpu_lb_stats, pcpu_lb_stats);
+
+static inline struct sd_lb_stats *this_sds(void)
+{
+	lockdep_assert_irqs_disabled();
+	return this_cpu_ptr(&pcpu_lb_stats.sds);
+}
+
+static inline struct sg_lb_stats *this_sgs(void)
+{
+	lockdep_assert_irqs_disabled();
+	return this_cpu_ptr(&pcpu_lb_stats.sgs);
+}
+
+static inline struct sg_lb_stats *this_local_sgs(void)
+{
+	lockdep_assert_irqs_disabled();
+	return this_cpu_ptr(&pcpu_lb_stats.local_sgs);
+}
+
+static inline struct sg_lb_stats *this_idlest_sgs(void)
+{
+	lockdep_assert_irqs_disabled();
+	return this_cpu_ptr(&pcpu_lb_stats.idlest_sgs);
+}
+
 static inline void init_sd_lb_stats(struct sd_lb_stats *sds)
 {
 	/*
@@ -12406,12 +12439,13 @@ static struct sched_group *
 sched_balance_find_dst_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 {
 	struct sched_group *idlest = NULL, *local = NULL, *group = sd->groups;
-	struct sg_lb_stats local_sgs, tmp_sgs;
+	struct sg_lb_stats *local_sgs = this_local_sgs();
 	struct sg_lb_stats *sgs;
 	unsigned long imbalance;
-	struct sg_lb_stats idlest_sgs = {
-			.avg_load = UINT_MAX,
-			.group_type = group_overloaded,
+	struct sg_lb_stats *idlest_sgs = this_idlest_sgs();
+	*idlest_sgs = (struct sg_lb_stats){
+		.avg_load = UINT_MAX,
+		.group_type = group_overloaded,
 	};
 
 	do {
@@ -12430,17 +12464,17 @@ sched_balance_find_dst_group(struct sched_domain *sd, struct task_struct *p, int
 					       sched_group_span(group));
 
 		if (local_group) {
-			sgs = &local_sgs;
+			sgs = local_sgs;
 			local = group;
 		} else {
-			sgs = &tmp_sgs;
+			sgs = this_sgs();
 		}
 
 		update_sg_wakeup_stats(sd, group, sgs, p);
 
-		if (!local_group && update_pick_idlest(idlest, &idlest_sgs, group, sgs)) {
+		if (!local_group && update_pick_idlest(idlest, idlest_sgs, group, sgs)) {
 			idlest = group;
-			idlest_sgs = *sgs;
+			*idlest_sgs = *sgs;
 		}
 
 	} while (group = group->next, group != sd->groups);
@@ -12458,17 +12492,17 @@ sched_balance_find_dst_group(struct sched_domain *sd, struct task_struct *p, int
 	 * If the local group is idler than the selected idlest group
 	 * don't try and push the task.
 	 */
-	if (local_sgs.group_type < idlest_sgs.group_type)
+	if (local_sgs->group_type < idlest_sgs->group_type)
 		return NULL;
 
 	/*
 	 * If the local group is busier than the selected idlest group
 	 * try and push the task.
 	 */
-	if (local_sgs.group_type > idlest_sgs.group_type)
+	if (local_sgs->group_type > idlest_sgs->group_type)
 		return idlest;
 
-	switch (local_sgs.group_type) {
+	switch (local_sgs->group_type) {
 	case group_overloaded:
 	case group_fully_busy:
 
@@ -12486,17 +12520,17 @@ sched_balance_find_dst_group(struct sched_domain *sd, struct task_struct *p, int
 		 */
 
 		if ((sd->flags & SD_NUMA) &&
-		    ((idlest_sgs.avg_load + imbalance) >= local_sgs.avg_load))
+		    ((idlest_sgs->avg_load + imbalance) >= local_sgs->avg_load))
 			return NULL;
 
 		/*
 		 * If the local group is less loaded than the selected
 		 * idlest group don't try and push any tasks.
 		 */
-		if (idlest_sgs.avg_load >= (local_sgs.avg_load + imbalance))
+		if (idlest_sgs->avg_load >= (local_sgs->avg_load + imbalance))
 			return NULL;
 
-		if (100 * local_sgs.avg_load <= sd->imbalance_pct * idlest_sgs.avg_load)
+		if (100 * local_sgs->avg_load <= sd->imbalance_pct * idlest_sgs->avg_load)
 			return NULL;
 		break;
 
@@ -12545,9 +12579,9 @@ sched_balance_find_dst_group(struct sched_domain *sd, struct task_struct *p, int
 				imb_numa_nr = min(w, sd->imb_numa_nr);
 			}
 
-			imbalance = abs(local_sgs.idle_cpus - idlest_sgs.idle_cpus);
+			imbalance = abs(local_sgs->idle_cpus - idlest_sgs->idle_cpus);
 			if (!adjust_numa_imbalance(imbalance,
-						   local_sgs.sum_nr_running + 1,
+						   local_sgs->sum_nr_running + 1,
 						   imb_numa_nr)) {
 				return NULL;
 			}
@@ -12560,7 +12594,7 @@ sched_balance_find_dst_group(struct sched_domain *sd, struct task_struct *p, int
 		 * up that the group has less spare capacity but finally more
 		 * idle CPUs which means more opportunity to run task.
 		 */
-		if (local_sgs.idle_cpus >= idlest_sgs.idle_cpus)
+		if (local_sgs->idle_cpus >= idlest_sgs->idle_cpus)
 			return NULL;
 		break;
 	}
@@ -12647,14 +12681,13 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd
 {
 	struct sched_group *sg = env->sd->groups;
 	struct sg_lb_stats *local = &sds->local_stat;
-	struct sg_lb_stats tmp_sgs;
 	unsigned long sum_util = 0;
 	bool sg_overloaded = 0, sg_overutilized = 0;
 
 	env->dst_core_idle = !sched_smt_active() || is_core_idle(env->dst_cpu);
 
 	do {
-		struct sg_lb_stats *sgs = &tmp_sgs;
+		struct sg_lb_stats *sgs = this_sgs();
 		int local_group;
 
 		local_group = cpumask_test_cpu(env->dst_cpu, sched_group_span(sg));
@@ -12929,21 +12962,21 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s
 static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 {
 	struct sg_lb_stats *local, *busiest;
-	struct sd_lb_stats sds;
+	struct sd_lb_stats *sds = this_sds();
 
-	init_sd_lb_stats(&sds);
+	init_sd_lb_stats(sds);
 
 	/*
 	 * Compute the various statistics relevant for load balancing at
 	 * this level.
 	 */
-	update_sd_lb_stats(env, &sds);
+	update_sd_lb_stats(env, sds);
 
 	/* There is no busy sibling group to pull tasks from */
-	if (!sds.busiest)
+	if (!sds->busiest)
 		goto out_balanced;
 
-	busiest = &sds.busiest_stat;
+	busiest = &sds->busiest_stat;
 
 	/* Misfit tasks should be dealt with regardless of the avg load */
 	if (busiest->group_type == group_misfit_task)
@@ -12965,7 +12998,7 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 	if (busiest->group_type == group_imbalanced)
 		goto force_balance;
 
-	local = &sds.local_stat;
+	local = &sds->local_stat;
 	/*
 	 * If the local group is busier than the selected busiest group
 	 * don't try and pull any tasks.
@@ -12986,14 +13019,14 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 			goto out_balanced;
 
 		/* XXX broken for overlapping NUMA groups */
-		sds.avg_load = (sds.total_load * SCHED_CAPACITY_SCALE) /
-				sds.total_capacity;
+		sds->avg_load = (sds->total_load * SCHED_CAPACITY_SCALE) /
+				sds->total_capacity;
 
 		/*
 		 * Don't pull any tasks if this group is already above the
 		 * domain average load.
 		 */
-		if (local->avg_load >= sds.avg_load)
+		if (local->avg_load >= sds->avg_load)
 			goto out_balanced;
 
 		/*
@@ -13009,9 +13042,9 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 	 * Try to move all excess tasks to a sibling domain of the busiest
 	 * group's child domain.
 	 */
-	if (sds.prefer_sibling && local->group_type == group_has_spare &&
+	if (sds->prefer_sibling && local->group_type == group_has_spare &&
 	    (busiest->group_type == group_llc_balance ||
-	    sibling_imbalance(env, &sds, busiest, local) > 1))
+	    sibling_imbalance(env, sds, busiest, local) > 1))
 		goto force_balance;
 
 	if (busiest->group_type != group_overloaded) {
@@ -13025,7 +13058,7 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 		}
 
 		if (busiest->group_type == group_smt_balance &&
-		    smt_vs_nonsmt_groups(sds.local, sds.busiest)) {
+		    smt_vs_nonsmt_groups(sds->local, sds->busiest)) {
 			/* Let non SMT CPU pull from SMT CPU sharing with sibling */
 			goto force_balance;
 		}
@@ -13054,8 +13087,8 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 
 force_balance:
 	/* Looks like there is an imbalance. Compute it */
-	calculate_imbalance(env, &sds);
-	return env->imbalance ? sds.busiest : NULL;
+	calculate_imbalance(env, sds);
+	return env->imbalance ? sds->busiest : NULL;
 
 out_balanced:
 	env->imbalance = 0;
@@ -13958,6 +13991,7 @@ static void sched_balance_domains(struct rq *rq, enum cpu_idle_type idle)
 
 		interval = get_sd_balance_interval(sd, busy);
 		if (time_after_eq(jiffies, sd->last_balance + interval)) {
+			guard(irqsave)();
 			if (sched_balance_rq(cpu, rq, sd, idle, &continue_balancing)) {
 				/*
 				 * The LBF_DST_PINNED logic could have changed
maintainer
#30 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to Sebastian Andrzej Siewior
Quoted 23 lines · Sebastian Andrzej Siewior
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines

arch/Kconfig config VMAP_STACK · line 1553

Quoted 3 lines
arch/Kconfig +3 -0
+
+	  The wakeup latency of tasks with reclaimed stacks may increase,
+	  especially while the system is under memory pressure.

It says *may* increase and on RT it _definitely_ will increase since there is a kworker involved not to mention the memory allocation itself. Anyway. This either needs to stay away from PREEMPT_RT or find a way to exclude at the very least mlock()ed tasks.
Did lockdep see this?

Quoted 2 lines

It should have. They're taking spinlock inside raw_spinlock and lockdep should very much warn about that by default.

Yes. My point was that this was hidden from lockdep.

Quoted 7 lines
Quoted 3 lines

If I understood the whole exercise correct then you have a kernel stack of two pages and in best case you can unmap and release the second page while the task is napping.

THREAD_SIZE_ORDER 2
THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)

Quoted 8 lines
Quoted 1 line

that makes for 4 pages.

Oh. I wasn't aware that we have 16kib stacks these days. But looking at it we have it now for over 10 years… Judging from 6538b8ea886e4 ("x86_64: expand kernel stack to 16K") it might be temporary and things are better now? Arm64 has a different story according to 845ad05ec31e0 ("arm64: Change kernel stack size to 16K"). Risc-V also mentions "for now" in 0cac21b02ba5f ("riscv: use 16KB kernel stack on 64-bit").

FWIW, I'm hoping to make the arm64 stack size configurable on the kernel cmdline with a view to changing the default back to 8k (assuming a 4k page size). With 8k, we currently see some overflows in direct reclaim (mirrored by a later stacktrace in this thread), but Suren has been looking at ways to improve that (e.g. by moving it into a separate kthread) which look promising.

Will

#31 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to Peter Zijlstra
Quoted 33 lines · Peter Zijlstra
Quoted 31 lines · Sebastian Andrzej Siewior

arch/Kconfig config VMAP_STACK · line 1541

Quoted 13 lines · David Stevens
arch/Kconfig +5 -0
index fa7507ac8e13..adb4a5957996 100644
@@ -1534,6 +1534,24 @@ config VMAP_STACK
      backing virtual mappings with real shadow memory, and KASAN_VMALLOC
      must be enabled.
 
+config HAVE_ARCH_RECLAIMABLE_STACK
+   def_bool n
+
+config RECLAIMABLE_STACK
+   default !PREEMPT_RT && !PROC_KCORE

This shouldn't default like this for RT. It either is useable or it is not.

Quoted 12 lines
arch/Kconfig +12 -0
+   bool "Allow stacks of some blocked threads to be reclaimed"
+   depends on VMAP_STACK && !STACK_GROWSUP
+   depends on HAVE_ARCH_RECLAIMABLE_STACK
+   depends on !DEBUG_STACK_USAGE
+   depends on !KASAN_VMALLOC # TODO: add support for this
+   depends on !DEBUG_KMEMLEAK # TODO: add support for this
+   help
+     Enable this to allow the unused portion of kernel stacks of most
+     blocked tasks to be reclaimed.
+
+     The wakeup latency of tasks with reclaimed stacks may increase,
+     especially while the system is under memory pressure.
Quoted 8 lines
Quoted 5 lines

It says *may* increase and on RT it _definitely_ will increase since there is a kworker involved not to mention the memory allocation itself. Anyway. This either needs to stay away from PREEMPT_RT or find a way to exclude at the very least mlock()ed tasks.
Did lockdep see this?

It should have. They're taking spinlock inside raw_spinlock and lockdep should very much warn about that by default.

It does trylocks on spinlocks and local locks, inside of alloc_pages_nolock_noprof(). Lockdep considers that safe.

On PREEMPT_RT, it is necessary to skip the call to alloc_pages_nolock_noprof() from under pi_lock, since with that configuration spin_trylock can end up needing to take pi_lock. But at least on !PREEMPT_RT, there is no risk of deadlock.

-David

maintainer
#32 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 5 lines · David Stevens
Quoted 2 lines

It should have. They're taking spinlock inside raw_spinlock and lockdep should very much warn about that by default.

It does trylocks on spinlocks and local locks, inside of alloc_pages_nolock_noprof(). Lockdep considers that safe.

Argh, that alloc_pages_nolock thing is a horror show.

Quoted 4 lines

On PREEMPT_RT, it is necessary to skip the call to alloc_pages_nolock_noprof() from under pi_lock, since with that configuration spin_trylock can end up needing to take pi_lock. But at least on !PREEMPT_RT, there is no risk of deadlock.

Yeah, which puts the lie to that horrific hack Alexei did. We should probably teach lockdep about spin_trylock() not being safe and see the house of cards crumble.

#33 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 8 lines · Peter Zijlstra
Quoted 4 lines

On PREEMPT_RT, it is necessary to skip the call to alloc_pages_nolock_noprof() from under pi_lock, since with that configuration spin_trylock can end up needing to take pi_lock. But at least on !PREEMPT_RT, there is no risk of deadlock.

Yeah, which puts the lie to that horrific hack Alexei did. We should probably teach lockdep about spin_trylock() not being safe and see the house of cards crumble.

Why is spin_trylock() unsafe on PREEMPT_RT? It's not intuitive since one can mutex_trylock() in interrupt context or under spinlock.

maintainer
#34 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines

On PREEMPT_RT, it is necessary to skip the call to alloc_pages_nolock_noprof() from under pi_lock, since with that configuration spin_trylock can end up needing to take pi_lock. But at least on !PREEMPT_RT, there is no risk of deadlock.

Quoted 6 lines
Quoted 3 lines

Yeah, which puts the lie to that horrific hack Alexei did. We should probably teach lockdep about spin_trylock() not being safe and see the house of cards crumble.

Why is spin_trylock() unsafe on PREEMPT_RT? It's not intuitive since one can mutex_trylock() in interrupt context or under spinlock.

We should not be able to do mutex_trylock() from interrupt context.

Because then we end up with an owner that is not a task, or if we account the interrupted task as the owner, idle that owns a sleeping lock, which also leads to problems, such as trying to boost idle and other such nonsense.

#35 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to Matthew Wilcox
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines

On PREEMPT_RT, it is necessary to skip the call to alloc_pages_nolock_noprof() from under pi_lock, since with that configuration spin_trylock can end up needing to take pi_lock. But at least on !PREEMPT_RT, there is no risk of deadlock.

Quoted 6 lines
Quoted 3 lines

Yeah, which puts the lie to that horrific hack Alexei did. We should probably teach lockdep about spin_trylock() not being safe and see the house of cards crumble.

Why is spin_trylock() unsafe on PREEMPT_RT? It's not intuitive since one can mutex_trylock() in interrupt context or under spinlock.

no, you shouldn't do mutex_trylock() in IRQ. There was something, I don't remember but that is the reason people use semaphores because it is okay to do down_trylock() in IRQ. It might be unlock path.

Sebastian

#36 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 9 lines · Matthew Wilcox
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines

On PREEMPT_RT, it is necessary to skip the call to alloc_pages_nolock_noprof() from under pi_lock, since with that configuration spin_trylock can end up needing to take pi_lock. But at least on !PREEMPT_RT, there is no risk of deadlock.

Yeah, which puts the lie to that horrific hack Alexei did. We should probably teach lockdep about spin_trylock() not being safe and see the house of cards crumble.

Is it already happening then?

https://lore.kernel.org/all/[email protected]/

Quoted 6 lines
Quoted 2 lines

Why is spin_trylock() unsafe on PREEMPT_RT? It's not intuitive since one can mutex_trylock() in interrupt context or under spinlock.

no, you shouldn't do mutex_trylock() in IRQ. There was something, I don't remember but that is the reason people use semaphores because it is okay to do down_trylock() in IRQ. It might be unlock path.

Is it what this comment (recently moved to can_spin_trylock()) explains?

/*
* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is
* unsafe in NMI. If spin_trylock() is called from hard IRQ the current
* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will
* mark the task as the owner of another rt_spin_lock which will
* confuse PI logic, so return immediately if called from hard IRQ or
* NMI.
*
* Note, irqs_disabled() case is ok. spin_trylock() can be called
* from raw_spin_lock_irqsave region.
*/
if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq()))
return false;

Of course checking for being under raw_spinlock_t can't be added like this, hm.

Quoted 1 line

Sebastian

#37 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 8 lines · Vlastimil Babka (SUSE)
Quoted 7 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines

On PREEMPT_RT, it is necessary to skip the call to alloc_pages_nolock_noprof() from under pi_lock, since with that configuration spin_trylock can end up needing to take pi_lock. But at least on !PREEMPT_RT, there is no risk of deadlock.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Yeah, which puts the lie to that horrific hack Alexei did. We should probably teach lockdep about spin_trylock() not being safe and see the house of cards crumble.

Is it already happening then?

https://lore.kernel.org/all/[email protected]/

here I would strip kfree_rcu_sheaf() from the kfree_rcu().

Quoted 23 lines
Quoted 6 lines
Quoted 2 lines

Why is spin_trylock() unsafe on PREEMPT_RT? It's not intuitive since one can mutex_trylock() in interrupt context or under spinlock.

no, you shouldn't do mutex_trylock() in IRQ. There was something, I don't remember but that is the reason people use semaphores because it is okay to do down_trylock() in IRQ. It might be unlock path.

Is it what this comment (recently moved to can_spin_trylock()) explains?

/*
* In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is
* unsafe in NMI. If spin_trylock() is called from hard IRQ the current
* task may be waiting for one rt_spin_lock, but rt_spin_trylock() will
* mark the task as the owner of another rt_spin_lock which will
* confuse PI logic, so return immediately if called from hard IRQ or
* NMI.
*
* Note, irqs_disabled() case is ok. spin_trylock() can be called
* from raw_spin_lock_irqsave region.
*/
if (IS_ENABLED(CONFIG_PREEMPT_RT) && (in_nmi() || in_hardirq()))
return false;

Quoted 2 lines

Of course checking for being under raw_spinlock_t can't be added like this, hm.

Yes.

Sebastian

#38 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

In reply to Sebastian Andrzej Siewior
Quoted 32 lines · Sebastian Andrzej Siewior

arch/Kconfig config VMAP_STACK · line 1541

Quoted 13 lines · David Stevens
arch/Kconfig +5 -0
index fa7507ac8e13..adb4a5957996 100644
@@ -1534,6 +1534,24 @@ config VMAP_STACK
        backing virtual mappings with real shadow memory, and KASAN_VMALLOC
        must be enabled.
 
+config HAVE_ARCH_RECLAIMABLE_STACK
+     def_bool n
+
+config RECLAIMABLE_STACK
+     default !PREEMPT_RT && !PROC_KCORE

This shouldn't default like this for RT. It either is useable or it is not.

arch/Kconfig config VMAP_STACK · line 1553

Quoted 12 lines
arch/Kconfig +12 -0
+     bool "Allow stacks of some blocked threads to be reclaimed"
+     depends on VMAP_STACK && !STACK_GROWSUP
+     depends on HAVE_ARCH_RECLAIMABLE_STACK
+     depends on !DEBUG_STACK_USAGE
+     depends on !KASAN_VMALLOC # TODO: add support for this
+     depends on !DEBUG_KMEMLEAK # TODO: add support for this
+     help
+       Enable this to allow the unused portion of kernel stacks of most
+       blocked tasks to be reclaimed.
+
+       The wakeup latency of tasks with reclaimed stacks may increase,
+       especially while the system is under memory pressure.
Quoted 2 lines

It says *may* increase and on RT it _definitely_ will increase since there is a kworker involved not to mention the memory allocation itself.

You're correct that this phrasing is wrong. A more accurate way to say it might be: "Under memory pressure, tasks that are not scheduled for extended periods may have their stacks reclaimed. Such tasks will experience increased wakeup latency."

Quoted 2 lines

Anyway. This either needs to stay away from PREEMPT_RT or find a way to exclude at the very least mlock()ed tasks.

Disabling kernel stack reclaim for mlockall() tasks definitely makes sense.

Quoted 1 line

Did lockdep see this?

What are you referring to by "this"? I did stress testing with lockdep enabled and didn't see any errors, although I will admit the bulk of that stress testing was !PREEMPT_RT.

Quoted 9 lines

If I understood the whole exercise correct then you have a kernel stack of two pages and in best case you can unmap and release the second page while the task is napping.

What might be a tad simpler is to memset(,0,) the remaining part of the stack. Since the stack is vmap-ed it should be swapped out on its own without additional tricks. That memset() would help zram to compress better so it uses less memory. ta-da.

Quoted 8 lines

What also should be simpler (and I am not saying just to move you away from the scheduler) is to have a shrinker which iterates over all tasks which are marked for reclaim and then similar to swap just unmap both stack pages and release the second page which is not used. Upon wake up the task should create a page_fault which would be used to allocate the second stack page and map the whole stack again.

This sounds simpler.

There is no swap for kernel memory, only user memory. I did contribute to some previous work that aimed to only only prepopulate one page of each kernel stack and then to dynamically fault in further pages as needed [1], but handling that architecturally and guaranteeing that memory is available when needed is difficult. Full-on swap to zram within the kernel would complicate pre-allocating pages for kernel stack faults further, since you could start getting inter-thread kernel stack faults due to blocked tasks putting pointers to their stacks into waitqueues and similar structures.

[1] https://lore.kernel.org/linux-mm/[email protected]/#r

#39 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 5 lines · David Stevens
Quoted 1 line

Did lockdep see this?

What are you referring to by "this"? I did stress testing with lockdep enabled and didn't see any errors, although I will admit the bulk of that stress testing was !PREEMPT_RT.

I am (was) just surprised that lockdep did not complain about the memory allocation under the PI lock. It was pointed out in that thread later that the new trylock usage could mask that.

Quoted 16 lines
Quoted 16 lines

If I understood the whole exercise correct then you have a kernel stack of two pages and in best case you can unmap and release the second page while the task is napping.

What might be a tad simpler is to memset(,0,) the remaining part of the stack. Since the stack is vmap-ed it should be swapped out on its own without additional tricks. That memset() would help zram to compress better so it uses less memory. ta-da.

What also should be simpler (and I am not saying just to move you away from the scheduler) is to have a shrinker which iterates over all tasks which are marked for reclaim and then similar to swap just unmap both stack pages and release the second page which is not used. Upon wake up the task should create a page_fault which would be used to allocate the second stack page and map the whole stack again.

Quoted 7 lines
Quoted 1 line

This sounds simpler.

There is no swap for kernel memory, only user memory. I did contribute to some previous work that aimed to only only prepopulate one page of each kernel stack and then to dynamically fault in further pages as needed [1], but handling that architecturally and guaranteeing that memory is available when needed is difficult. Full-on swap to zram

but faulting it in while the task is running is sort of complicated since the task might have page_faults disabled. This is what I missed initially (the task is with disabled interrupts in the scheduler).

But unmap the stack while application is sleeping (via shrinker) and fault it in via another thread on wake up should be doable.

Sebastian

#40 lore raw

Re: [RFC 06/10] Reclaim memory from blocked kernel stacks

Quoted 11 lines · Sebastian Andrzej Siewior
Quoted 5 lines · David Stevens
Quoted 1 line

Did lockdep see this?

What are you referring to by "this"? I did stress testing with lockdep enabled and didn't see any errors, although I will admit the bulk of that stress testing was !PREEMPT_RT.

I am (was) just surprised that lockdep did not complain about the memory allocation under the PI lock. It was pointed out in that thread later that the new trylock usage could mask that.

Lockdep did complain on PREEMPT_RT, since spinlocks can take pi_lock. That's why PREEMPT_RT always defers repopulating the stack in this series.

The trylock usage is sufficient to prevent any lockdep issues on !PREEMPT_RT, however.

Quoted 1 line
Quoted 1 line
Quoted 1 line

If I understood the whole exercise correct then you have a kernel

stack

Quoted 1 line
Quoted 1 line
Quoted 1 line

of two pages and in best case you can unmap and release the second

page

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

while the task is napping.

What might be a tad simpler is to memset(,0,) the remaining part of

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

stack. Since the stack is vmap-ed it should be swapped out on its own without additional tricks. That memset() would help zram to compress better so it uses less memory. ta-da.

What also should be simpler (and I am not saying just to move you away from the scheduler) is to have a shrinker which iterates over all

tasks

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

which are marked for reclaim and then similar to swap just unmap both stack pages and release the second page which is not used. Upon wake up the task should create a page_fault which would be used

to

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

allocate the second stack page and map the whole stack again.

This sounds simpler.

There is no swap for kernel memory, only user memory. I did contribute to some previous work that aimed to only only prepopulate one page of each kernel stack and then to dynamically fault in further pages as needed [1], but handling that architecturally and guaranteeing that memory is available when needed is difficult. Full-on swap to zram

Quoted 6 lines

but faulting it in while the task is running is sort of complicated since the task might have page_faults disabled. This is what I missed initially (the task is with disabled interrupts in the scheduler).

But unmap the stack while application is sleeping (via shrinker) and fault it in via another thread on wake up should be doable.

Faulting on another thread's stack does bypass the architectural problems of faulting on your own stack. However, since the faults could happen in an atomic context, it has the same problem of panicing on memory allocation failure. GFP_ATOMIC obviously wouldn't work. It would be possible to have a pool of pre-allocated pages to handle these faults, but I can't think of any way to size the pool to guarantee it never runs out.

-David

#41 lore raw

RFC 7/10 Reclaim stacks via a shrinker

In reply to David Stevens

Use a shrinker to reclaim unused portions of blocked task stacks. The core of the shrinker is a list_lru. A referenced bit is used to require that a stack remains unused for long enough to be seen by the shrinker twice.

To minimize how much work is done in the scheduler path and under the pi_lock, tasks are initially put onto a regular list before being moved onto the list_lru by a shrinker. Additionally, tasks are not proactively removed from the lru when they become unblocked - again, to reduce what happens in the scheduler path.

---
 include/linux/sched.h   |  13 +-
 kernel/stack_shrinker.c | 400 ++++++++++++++++++++++++++++++++++------
 2 files changed, 357 insertions(+), 56 deletions(-)
include/linux/sched.h +9 -4
index c93a234fac96..dc241d1c058f 100644
@@ -83,7 +83,6 @@ struct sched_dl_entity;
 struct seq_file;
 struct sighand_struct;
 struct signal_struct;
-struct stack_reclaim_work;
 struct task_delay_info;
 struct task_exec_state;
 struct task_group;
@@ -835,9 +834,17 @@ enum stack_reclaim_enum {
 	STACK_RECLAIMED		= 5,
 } __packed;
 
+enum stack_reclaim_lru_state_enum {
+	STACK_LRU_NOT_PRESENT	= 0,
+	STACK_LRU_PENDING	= 1,
+	STACK_LRU_NEW		= 2,
+	STACK_LRU_OLD		= 3,
+} __packed;
+
 union stack_reclaim_state {
 	struct {
 		enum stack_reclaim_enum stack_state;
+		enum stack_reclaim_lru_state_enum lru_state;
 		u16 node;
 	};
 	u32 val;
@@ -845,6 +852,7 @@ union stack_reclaim_state {
 
 union stack_reclaim_list {
 	struct llist_node refill_entry;
+	struct list_head reclaim_entry;
 };
 #endif
 
@@ -1618,9 +1626,6 @@ struct task_struct {
 #ifdef CONFIG_MEMCG
 	struct obj_cgroup		*stack_obj_cgroup;
 #endif
-
-	// TODO: Replace these with a shrinker
-	struct stack_reclaim_work	*stack_reclaim_work;
 #endif
 #endif
 #ifdef CONFIG_THREAD_INFO_IN_TASK
kernel/stack_shrinker.c +348 -52

485 lines not on this page · raw message

Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#42 lore raw

RFC 8/10 Set PF_RECLAIMABLE_STACK in various places

In reply to David Stevens

Annotate various blocking locations with the PF_RECLAIMABLE_STACK.

---
 drivers/android/binder/thread.rs | 14 +++++++++++++
 fs/eventpoll.c                   |  3 +++
 fs/pipe.c                        | 28 ++++++++++++++++---------
 fs/select.c                      |  3 +++
 kernel/futex/waitwake.c          |  3 +++
 kernel/signal.c                  | 36 +++++++++++++++++++-------------
 kernel/time/hrtimer.c            |  3 +++
 rust/kernel/task.rs              | 16 ++++++++++++++
 8 files changed, 82 insertions(+), 24 deletions(-)
drivers/android/binder/thread.rs +14 -0
index bc0ef8927905..be62f7fd43ca 100644
@@ -538,13 +538,21 @@ fn get_work_local(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn Deliv
 
         // Loop waiting only on the local queue (i.e., not registering with the process queue).
         let mut inner = self.inner.lock();
+        // SAFETY: Only accessed locally in this function
+        let current = unsafe { Task::current() };
         loop {
             if let Some(work) = inner.pop_work() {
                 return Ok(Some(work));
             }
 
             inner.looper_flags |= LOOPER_WAITING;
+
+            current.set_flag_bits(bindings::PF_RECLAIMABLE_STACK);
+
             let signal_pending = self.work_condvar.wait_interruptible_freezable(&mut inner);
+
+            current.clear_flag_bits(bindings::PF_RECLAIMABLE_STACK);
+
             inner.looper_flags &= !LOOPER_WAITING;
 
             if signal_pending {
@@ -592,15 +600,21 @@ fn get_work(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn DeliverToRe
         };
 
         let mut inner = self.inner.lock();
+        // SAFETY: Only accessed locally in this function
+        let current = unsafe { Task::current() };
         loop {
             if let Some(work) = inner.pop_work() {
                 return Ok(Some(work));
             }
 
+            current.set_flag_bits(bindings::PF_RECLAIMABLE_STACK);
+
             inner.looper_flags |= LOOPER_WAITING | LOOPER_WAITING_PROC;
             let signal_pending = self.work_condvar.wait_interruptible_freezable(&mut inner);
             inner.looper_flags &= !(LOOPER_WAITING | LOOPER_WAITING_PROC);
 
+            current.clear_flag_bits(bindings::PF_RECLAIMABLE_STACK);
+
             if signal_pending || inner.looper_need_return {
                 // We need to return now. We need to pull the thread off the list of ready threads
                 // (by dropping `reg`), then check the state again after it's off the list to
fs/eventpoll.c +3 -0
index eed8cecd94e3..4e9c09446b48 100644
@@ -9,6 +9,7 @@
 #include <linux/init.h>
 #include <linux/kernel.h>
 #include <linux/sched/signal.h>
+#include <linux/sched/task_stack.h>
 #include <linux/fs.h>
 #include <linux/file.h>
 #include <linux/signal.h>
@@ -2302,6 +2303,8 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events,
 		if (signal_pending(current))
 			return -EINTR;
 
+		guard(allow_stack_reclaim)();
+
 		/*
 		 * Internally init_wait() uses autoremove_wake_function(),
 		 * thus wait entry is removed from the wait queue on each
fs/pipe.c +18 -10
index 429b0714ec57..15503b12fe8f 100644
@@ -27,6 +27,7 @@
 #include <linux/watch_queue.h>
 #include <linux/sysctl.h>
 #include <linux/sort.h>
+#include <linux/sched/task_stack.h>
 
 #include <linux/uaccess.h>
 #include <asm/ioctls.h>
@@ -469,16 +470,23 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
 			break;
 		}
 		mutex_unlock(&pipe->mutex);
-		/*
-		 * We only get here if we didn't actually read anything.
-		 *
-		 * But because we didn't read anything, at this point we can
-		 * just return directly with -ERESTARTSYS if we're interrupted,
-		 * since we've done any required wakeups and there's no need
-		 * to mark anything accessed. And we've dropped the lock.
-		 */
-		if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
-			return -ERESTARTSYS;
+
+		{
+			guard(allow_stack_reclaim)();
+			/*
+			 * We only get here if we didn't actually read
+			 * anything.
+			 *
+			 * But because we didn't read anything, at this point
+			 * we can just return directly with -ERESTARTSYS if
+			 * we're interrupted, since we've done any required
+			 * wakeups and there's no need to mark anything
+			 * accessed. And we've dropped the lock.
+			 */
+			if (wait_event_interruptible_exclusive(pipe->rd_wait,
+							       pipe_readable(pipe)) < 0)
+				return -ERESTARTSYS;
+		}
 
 		wake_next_reader = true;
 		mutex_lock(&pipe->mutex);
fs/select.c +3 -0
index 95d76531015a..3af1bf4d74a9 100644
@@ -19,6 +19,7 @@
 #include <linux/kernel.h>
 #include <linux/sched/signal.h>
 #include <linux/sched/rt.h>
+#include <linux/sched/task_stack.h>
 #include <linux/syscalls.h>
 #include <linux/export.h>
 #include <linux/slab.h>
@@ -236,6 +237,8 @@ static int poll_schedule_timeout(struct poll_wqueues *pwq, int state,
 {
 	int rc = -EINTR;
 
+	guard(allow_stack_reclaim)();
+
 	set_current_state(state);
 	if (!READ_ONCE(pwq->triggered))
 		rc = schedule_hrtimeout_range(expires, slack, HRTIMER_MODE_ABS);
kernel/futex/waitwake.c +3 -0
index d4483d15d30a..0fbf7bfcd904 100644
@@ -2,6 +2,7 @@
 
 #include <linux/plist.h>
 #include <linux/sched/task.h>
+#include <linux/sched/task_stack.h>
 #include <linux/sched/signal.h>
 #include <linux/freezer.h>
 
@@ -378,6 +379,7 @@ void futex_do_wait(struct futex_q *q, struct hrtimer_sleeper *timeout)
 	 * has tried to wake us, and we can skip the call to schedule().
 	 */
 	if (likely(!plist_node_empty(&q->list))) {
+		guard(allow_stack_reclaim)();
 		/*
 		 * If the timer has already expired, current will already be
 		 * flagged for rescheduling. Only call schedule if there
@@ -547,6 +549,7 @@ static void futex_sleep_multiple(struct futex_vector *vs, unsigned int count,
 			return;
 	}
 
+	guard(allow_stack_reclaim)();
 	schedule();
 }
 
kernel/signal.c +22 -14
index bbc0fd4cc4d7..ca0d79ee013f 100644
@@ -2718,17 +2718,21 @@ static void do_freezer_trap(void)
 		return;
 	}
 
-	/*
-	 * Now we're sure that there is no pending fatal signal and no
-	 * pending traps. Clear TIF_SIGPENDING to not get out of schedule()
-	 * immediately (if there is a non-fatal signal pending), and
-	 * put the task into sleep.
-	 */
-	__set_current_state(TASK_INTERRUPTIBLE|TASK_FREEZABLE);
-	clear_thread_flag(TIF_SIGPENDING);
-	spin_unlock_irq(&current->sighand->siglock);
-	cgroup_enter_frozen();
-	schedule();
+	{
+		guard(allow_stack_reclaim)();
+
+		/*
+		 * Now we're sure that there is no pending fatal signal and no
+		 * pending traps. Clear TIF_SIGPENDING to not get out of schedule()
+		 * immediately (if there is a non-fatal signal pending), and
+		 * put the task into sleep.
+		 */
+		__set_current_state(TASK_INTERRUPTIBLE | TASK_FREEZABLE);
+		clear_thread_flag(TIF_SIGPENDING);
+		spin_unlock_irq(&current->sighand->siglock);
+		cgroup_enter_frozen();
+		schedule();
+	}
 
 	/*
 	 * We could've been woken by task_work, run it to clear
@@ -3788,9 +3792,13 @@ static int do_sigtimedwait(const sigset_t *which, kernel_siginfo_t *info,
 		recalc_sigpending();
 		spin_unlock_irq(&tsk->sighand->siglock);
 
-		__set_current_state(TASK_INTERRUPTIBLE|TASK_FREEZABLE);
-		ret = schedule_hrtimeout_range(to, tsk->timer_slack_ns,
-					       HRTIMER_MODE_REL);
+		{
+			guard(allow_stack_reclaim)();
+			__set_current_state(TASK_INTERRUPTIBLE | TASK_FREEZABLE);
+			ret = schedule_hrtimeout_range(to, tsk->timer_slack_ns,
+						       HRTIMER_MODE_REL);
+		}
+
 		spin_lock_irq(&tsk->sighand->siglock);
 		__set_task_blocked(tsk, &tsk->real_blocked);
 		sigemptyset(&tsk->real_blocked);
kernel/time/hrtimer.c +3 -0
index 313dcea127fe..30c85482dfd5 100644
@@ -39,6 +39,7 @@
 #include <linux/sched/nohz.h>
 #include <linux/sched/debug.h>
 #include <linux/sched/isolation.h>
+#include <linux/sched/task_stack.h>
 #include <linux/timer.h>
 #include <linux/freezer.h>
 #include <linux/compat.h>
@@ -2392,6 +2393,8 @@ static int __sched do_nanosleep(struct hrtimer_sleeper *t, enum hrtimer_mode mod
 	struct restart_block *restart;
 
 	do {
+		guard(allow_stack_reclaim)();
+
 		set_current_state(TASK_INTERRUPTIBLE|TASK_FREEZABLE);
 		hrtimer_sleeper_start_expires(t, mode);
 
rust/kernel/task.rs +16 -0
index 38273f4eedb5..6168f058343a 100644
@@ -344,6 +344,22 @@ pub fn group_leader(&self) -> &Task {
         // only be used while `current` is still valid, thus still running.
         unsafe { &*ptr.cast() }
     }
+
+    /// Sets the given task flag bits on the current task.
+    #[inline]
+    pub fn set_flag_bits(&self, set: u32) {
+            // SAFETY: The `flags` field of `current` is not modified from other threads, so
+	    // the non-atomic update isn't a race.
+	    unsafe { (*self.as_ptr()).flags |= set }
+    }
+
+    /// Clears the given task flag bits on the current task.
+    #[inline]
+    pub fn clear_flag_bits(&self, clear: u32) {
+            // SAFETY: The `flags` field of `current` is not modified from other threads, so
+	    // the non-atomic update isn't a race.
+	    unsafe { (*self.as_ptr()).flags &= !clear }
+    }
 }
 
 // SAFETY: The type invariants guarantee that `Task` is always refcounted.
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#43 lore raw

Re: [RFC 08/10] Set PF_RECLAIMABLE_STACK in various places

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Safe Rust methods `set_flag_bits` and `clear_flag_bits` on `Task` allow data races when called on…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Safe Rust methods set_flag_bits and clear_flag_bits on Task allow data races when called on non-current tasks.
- [Medium] Unconditional clearing of PF_RECLAIMABLE_STACK in Rust binder breaks flag preservation logic. --

commit 6f8ac02f62f897200c71ad824d4019095dc4428c
Author: David Stevens <[email protected]>

Set PF_RECLAIMABLE_STACK in various places

Annotate various blocking locations with the PF_RECLAIMABLE_STACK.

Quoted 5 lines
drivers/android/binder/thread.rs +0 -0
index bc0ef8927905c..be62f7fd43ca2 100644
@@ -538,13 +538,21 @@ fn get_work_local(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn Deliv

[ ... ]

drivers/android/binder/thread.rs fn get_work_local(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn Deliv · line 554

Quoted 7 lines
             inner.looper_flags |= LOOPER_WAITING;
drivers/android/binder/thread.rs +5 -0
+
+            current.set_flag_bits(bindings::PF_RECLAIMABLE_STACK);
+
             let signal_pending = self.work_condvar.wait_interruptible_freezable(&mut inner);
+
+            current.clear_flag_bits(bindings::PF_RECLAIMABLE_STACK);

[Severity: Medium]
Will unconditionally clearing the flag break flag preservation logic if stack reclaim states are nested?

The C allow_stack_reclaim guard explicitly saves and restores the flag to support nesting. If get_work_local() is called from a context where PF_RECLAIMABLE_STACK is already set, this will unexpectedly disable stack reclamation for the outer scope upon exit.

Quoted 5 lines
drivers/android/binder/thread.rs +1 -0
+
             inner.looper_flags &= !LOOPER_WAITING;
 
             if signal_pending {
@@ -592,15 +600,21 @@ fn get_work(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn DeliverToRe

[ ... ]

drivers/android/binder/thread.rs fn get_work(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn DeliverToRe · line 616

Quoted 7 lines
drivers/android/binder/thread.rs +3 -0
+            current.set_flag_bits(bindings::PF_RECLAIMABLE_STACK);
+
             inner.looper_flags |= LOOPER_WAITING | LOOPER_WAITING_PROC;
             let signal_pending = self.work_condvar.wait_interruptible_freezable(&mut inner);
             inner.looper_flags &= !(LOOPER_WAITING | LOOPER_WAITING_PROC);
 
+            current.clear_flag_bits(bindings::PF_RECLAIMABLE_STACK);

[Severity: Medium]
Similar to get_work_local(), is it safe to unconditionally clear the flag here in get_work() instead of restoring its previous state?

Quoted 10 lines
drivers/android/binder/thread.rs +1 -0
+
             if signal_pending || inner.looper_need_return {
                 // We need to return now. We need to pull the thread off the list of ready threads
                 // (by dropping `reg`), then check the state again after it's off the list to
rust/kernel/task.rs +0 -0
index 38273f4eedb51..6168f058343a2 100644
@@ -344,6 +344,22 @@ pub fn group_leader(&self) -> &Task {
         // only be used while `current` is still valid, thus still running.

rust/kernel/task.rs pub fn group_leader(&self) -> &Task { · line 354

Quoted 10 lines
rust/kernel/task.rs +8 -0
         unsafe { &*ptr.cast() }
     }
+
+    /// Sets the given task flag bits on the current task.
+    #[inline]
+    pub fn set_flag_bits(&self, set: u32) {
+            // SAFETY: The `flags` field of `current` is not modified from other threads, so
+	    // the non-atomic update isn't a race.
+	    unsafe { (*self.as_ptr()).flags |= set }
+    }

[Severity: High]
Does this expose a soundness hole by allowing data races on the flags field when called on a non-current task?

The safety comment assumes self is the current task, but the method signature accepts any Task reference. Because Task implements Send + Sync, another thread could obtain a reference to a non-current task (for example, via group_leader()) and safely call this method. This allows an unsafe non-atomic read-modify-write and undefined behavior in safe Rust.

rust/kernel/task.rs pub fn group_leader(&self) -> &Task { · line 363

Quoted 9 lines
rust/kernel/task.rs +8 -0
+
+    /// Clears the given task flag bits on the current task.
+    #[inline]
+    pub fn clear_flag_bits(&self, clear: u32) {
+            // SAFETY: The `flags` field of `current` is not modified from other threads, so
+	    // the non-atomic update isn't a race.
+	    unsafe { (*self.as_ptr()).flags &= !clear }
+    }
 }

[Severity: High]
Does this method suffer from the same soundness hole as set_flag_bits() since it is a safe method performing a non-atomic update without verifying self is the current task?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
#44 lore raw

Re: [RFC 08/10] Set PF_RECLAIMABLE_STACK in various places

In reply to David Stevens

Hello David,

Quoted 7 lines · David Stevens
fs/pipe.c +0 -3
@@ -469,16 +470,23 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to)
                        break;
                }
                mutex_unlock(&pipe->mutex);
-               /*
-                * We only get here if we didn't actually read anything.
-                *

fs/pipe.c anon_pipe_read(struct kiocb *iocb, struct iov_iter *to) · line 475

Quoted 10 lines
fs/pipe.c +3 -7
-                * But because we didn't read anything, at this point we can
-                * just return directly with -ERESTARTSYS if we're interrupted,
-                * since we've done any required wakeups and there's no need
-                * to mark anything accessed. And we've dropped the lock.
-                */
-               if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe)) < 0)
-                       return -ERESTARTSYS;
+
+               {
+                       guard(allow_stack_reclaim)();

nit. You can just use a

    scoped_guard(allow_stack_reclaim) {
        if (wait_event_interruptible_exclusive(...))
            return -ERESTARTSYS;
    }

here.

Same comment for rest of the thread where a scoped_guard() can be used instead of this pattern.

Quoted 14 lines
fs/pipe.c +14 -0
+                       /*
+                        * We only get here if we didn't actually read
+                        * anything.
+                        *
+                        * But because we didn't read anything, at this point
+                        * we can just return directly with -ERESTARTSYS if
+                        * we're interrupted, since we've done any required
+                        * wakeups and there's no need to mark anything
+                        * accessed. And we've dropped the lock.
+                        */
+                       if (wait_event_interruptible_exclusive(pipe->rd_wait,
+                                                              pipe_readable(pipe)) < 0)
+                               return -ERESTARTSYS;
+               }
Signature · 2 lines
Thanks and Regards,
Prateek
maintainer
#45 lore raw

Re: [RFC 08/10] Set PF_RECLAIMABLE_STACK in various places

In reply to David Stevens
Quoted 19 lines · David Stevens

Annotate various blocking locations with the PF_RECLAIMABLE_STACK.

---
 drivers/android/binder/thread.rs | 14 +++++++++++++
 fs/eventpoll.c                   |  3 +++
 fs/pipe.c                        | 28 ++++++++++++++++---------
 fs/select.c                      |  3 +++
 kernel/futex/waitwake.c          |  3 +++
 kernel/signal.c                  | 36 +++++++++++++++++++-------------
 kernel/time/hrtimer.c            |  3 +++
 rust/kernel/task.rs              | 16 ++++++++++++++
 8 files changed, 82 insertions(+), 24 deletions(-)
drivers/android/binder/thread.rs +0 -0
index bc0ef8927905..be62f7fd43ca 100644

drivers/android/binder/thread.rs fn get_work_local(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn Deliv · line 542

Quoted 5 lines
drivers/android/binder/thread.rs +2 -0
@@ -538,13 +538,21 @@ fn get_work_local(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn Deliv
 
         // Loop waiting only on the local queue (i.e., not registering with the process queue).
         let mut inner = self.inner.lock();
+        // SAFETY: Only accessed locally in this function
+        let current = unsafe { Task::current() };

You can invoke kernel::current!() instead to avoid unsafe here.

drivers/android/binder/thread.rs fn get_work_local(self: &Arc<Self>, wait: bool) -> Result<Option<DLArc<dyn Deliv · line 554

Quoted 12 lines
drivers/android/binder/thread.rs +5 -0
         loop {
             if let Some(work) = inner.pop_work() {
                 return Ok(Some(work));
             }
 
             inner.looper_flags |= LOOPER_WAITING;
+
+            current.set_flag_bits(bindings::PF_RECLAIMABLE_STACK);
+
             let signal_pending = self.work_condvar.wait_interruptible_freezable(&mut inner);
+
+            current.clear_flag_bits(bindings::PF_RECLAIMABLE_STACK);

It would be nicer to define a guard type like you did in C so that this is less "manual".

Quoted 14 lines
rust/kernel/task.rs +6 -0
index 38273f4eedb5..6168f058343a 100644
@@ -344,6 +344,22 @@ pub fn group_leader(&self) -> &Task {
         // only be used while `current` is still valid, thus still running.
         unsafe { &*ptr.cast() }
     }
+
+    /// Sets the given task flag bits on the current task.
+    #[inline]
+    pub fn set_flag_bits(&self, set: u32) {
+            // SAFETY: The `flags` field of `current` is not modified from other threads, so
+	    // the non-atomic update isn't a race.

rust/kernel/task.rs pub fn group_leader(&self) -> &Task { · line 362

Quoted 10 lines
rust/kernel/task.rs +10 -0
+	    unsafe { (*self.as_ptr()).flags |= set }
+    }
+
+    /// Clears the given task flag bits on the current task.
+    #[inline]
+    pub fn clear_flag_bits(&self, clear: u32) {
+            // SAFETY: The `flags` field of `current` is not modified from other threads, so
+	    // the non-atomic update isn't a race.
+	    unsafe { (*self.as_ptr()).flags &= !clear }
+    }

It looks like the indentation here is a bit off. Please try invoking 'make rustfmt' to reformat this code.

Alice

#46 lore raw

RFC 9/10 x86: Enable reclaimable stacks

In reply to David Stevens

Implement top_of_blocked_task_stack() to get the stack pointer out of a thread_struct and enable HAVE_ARCH_RECLAIMABLE_STACK.

---
 arch/x86/Kconfig                 | 1 +
 arch/x86/include/asm/processor.h | 5 +++++
 2 files changed, 6 insertions(+)
arch/x86/Kconfig +1 -0
index bdad90f210e4..652d5a9c0830 100644
@@ -211,6 +211,7 @@ config X86
 	select HAVE_ARCH_USERFAULTFD_WP         if X86_64 && USERFAULTFD
 	select HAVE_ARCH_USERFAULTFD_MINOR	if X86_64 && USERFAULTFD
 	select HAVE_ARCH_VMAP_STACK		if X86_64
+	select HAVE_ARCH_RECLAIMABLE_STACK	if X86_64
 	select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET
 	select HAVE_ARCH_WITHIN_STACK_FRAMES
 	select HAVE_ASM_MODVERSIONS
arch/x86/include/asm/processor.h +5 -0
index 87b1d4c0727e..207e7adced64 100644
@@ -690,6 +690,11 @@ extern void start_thread(struct pt_regs *regs, unsigned long new_ip,
 #define GET_TSC_CTL(adr)	get_tsc_mode((adr))
 #define SET_TSC_CTL(val)	set_tsc_mode((val))
 
+static inline unsigned long top_of_blocked_task_stack(struct thread_struct *thread)
+{
+	return thread->sp;
+}
+
 extern int get_tsc_mode(unsigned long adr);
 extern int set_tsc_mode(unsigned int val);
 
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
#47 lore raw

RFC 10/10 arm64: Enable reclaimable stacks

In reply to David Stevens

Implement top_of_blocked_task_stack() to get the stack pointer out of a thread_struct and enable HAVE_ARCH_RECLAIMABLE_STACK.

---
 arch/arm64/Kconfig                 | 1 +
 arch/arm64/include/asm/processor.h | 5 +++++
 2 files changed, 6 insertions(+)
arch/arm64/Kconfig +1 -0
index b3afe0688919..262a4fddce19 100644
@@ -173,6 +173,7 @@ config ARM64
 	select HAVE_ARCH_TRACEHOOK
 	select HAVE_ARCH_TRANSPARENT_HUGEPAGE
 	select HAVE_ARCH_VMAP_STACK
+	select HAVE_ARCH_RECLAIMABLE_STACK
 	select HAVE_ARM_SMCCC
 	select HAVE_ASM_MODVERSIONS
 	select HAVE_EBPF_JIT
arch/arm64/include/asm/processor.h +5 -0
index c2a627f39314..f5a550835c94 100644
@@ -204,6 +204,11 @@ struct thread_struct {
 #endif
 };
 
+static inline unsigned long top_of_blocked_task_stack(struct thread_struct *thread)
+{
+	return thread->cpu_context.sp;
+}
+
 static inline unsigned int thread_get_vl(struct thread_struct *thread,
 					 enum vec_type type)
 {
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog
maintainer
#48 lore raw
In reply to David Stevens
Quoted 2 lines · David Stevens

On Android, system processes typically have 2000-3000 threads. App processes add 1000s more threads on top of this.

WTF ?!? Why does that all spawn *that* many threads? Perhaps work on reducing that some?

Quoted 9 lines

Tracking blocked state and when it is safe to reclaim a stack is done via a series of hooks in the scheduler. The actual reclaim of stacks is done asynchronously in a shrinker.

Once a task's stack has been reclaimed, it cannot be rescheduled until its stack is repopulated. Although there can be a repopulation fast path within the scheduler, reliably allocating memory to repopulate the stack requires a fallback path that defers the repopulation and wakeup to a workqueue context that can use GFP_KERNEL.

I am really confused. On the one hand you have John working on proxy execution, with the aim on reducing latencies, and then here you are, posting something that will introduce basically unbound latencies.

maintainer
#49 lore raw
Quoted 5 lines · On Fri, 28 Aug 2026 14:47:58 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines

On Android, system processes typically have 2000-3000 threads. App processes add 1000s more threads on top of this.

WTF ?!? Why does that all spawn *that* many threads? Perhaps work on reducing that some?

I guess you never ran Java ;-)

-- Steve

maintainer
#50 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Fri, 28 Aug 2026 14:47:58 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines

On Android, system processes typically have 2000-3000 threads. App processes add 1000s more threads on top of this.

WTF ?!? Why does that all spawn *that* many threads? Perhaps work on reducing that some?

I guess you never ran Java ;-)

Of course not; why would you want to do something that silly ;-)

maintainer
#51 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Steven Rostedt
Quoted 3 lines · On Fri, 28 Aug 2026 14:47:58 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 2 lines

On Android, system processes typically have 2000-3000 threads. App processes add 1000s more threads on top of this.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

WTF ?!? Why does that all spawn *that* many threads? Perhaps work on reducing that some?

I guess you never ran Java ;-)

Of course not; why would you want to do something that silly ;-)

I realized I actually have a Java thing, the kids have Minecraft, so I booted that up and I found all of 108 threads. Which is still kinda insane, but nowhere near the 1000s per app as claimed.

Reducing the thread count from O(1e3) to O(1e2) would win far more memory than any thread stack shenanigans would.

maintainer
#52 lore raw
Quoted 3 lines · On Fri, 28 Aug 2026 16:45:00 +0200 Peter Zijlstra <[email protected]> wrote:

I realized I actually have a Java thing, the kids have Minecraft, so I booted that up and I found all of 108 threads. Which is still kinda insane, but nowhere near the 1000s per app as claimed.

Have you looked at how many threads Chome uses?

 $ ps -eLf|grep chrome | wc -l
 930

Of course I probably have 900 tabs open :-p

-- Steve

#53 lore raw
In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 2 lines · David Stevens

On Android, system processes typically have 2000-3000 threads. App processes add 1000s more threads on top of this.

WTF ?!? Why does that all spawn *that* many threads? Perhaps work on reducing that some?

I agree that the number of threads is rather excessive, and reducing it in userspace is an ongoing effort. However, given Android's use of Java and its heavily multi-process architecture, especially once apps are involved, there are limits to how much the thread count can realistically be reduced. And even if a 50% reduction in thread count were somehow achieved, kernel stacks would still consume upwards of 1% of system RAM on lower spec devices with 4GB of memory.

Quoted 10 lines
Quoted 9 lines

Tracking blocked state and when it is safe to reclaim a stack is done via a series of hooks in the scheduler. The actual reclaim of stacks is done asynchronously in a shrinker.

Once a task's stack has been reclaimed, it cannot be rescheduled until its stack is repopulated. Although there can be a repopulation fast path within the scheduler, reliably allocating memory to repopulate the stack requires a fallback path that defers the repopulation and wakeup to a workqueue context that can use GFP_KERNEL.

Quoted 3 lines

I am really confused. On the one hand you have John working on proxy execution, with the aim on reducing latencies, and then here you are, posting something that will introduce basically unbound latencies.

The two projects aren't contradictory because they deal with different types of latency.

Proxy execution aims to solve latency introduced by priority inversion between background tasks and foreground tasks. This can happen at basically any point if you get unlucky, resulting in unpredictable latency spikes in high priority tasks.

The latency introduced by reclaimable stacks isn't substantially different from the latency from refaulting evicted anon or file pages. It's a tradeoff between the reduced cost of cold memory and the increased latency when re-accessing that cold memory - clearly a tradeoff that Linux already makes. In situations where reclaimed stacks introduce latency, there will almost certainly already be extra latency incurred from refaulting userspace pages. I don't yet have production data, but the testing I've done so far doesn't show any measurable negative impact on user visible latency metrics.

-David