RFC Reclaimable kernel stacks
Get series with b4
b4 am [email protected]Versions newest first
- v1 RFC Latest10 patches 41 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/10 | Add !MEMCG memcg_list_lru_alloc implementationinclude/ | +6 | 0 replies | |
| 2/10 | mm/vmalloc: Skip vmallocinfo NUMA stats for VM_SPARSEmm/ | +3 | 0 replies | |
| 3/10 | fork: refactor vmap stack alloc/free into helperskernel/ | +27 | 0 replies | |
| 4/10 | mm: vmalloc: support creating aligned vm areasinclude/ mm/ | +25 | 0 replies | |
| 5/10 | fork: allocate reclaimable stacks with VM_SPARSEkernel/ | +92 | 0 replies | |
| 6/10 | Reclaim memory from blocked kernel stacksarch/ include/ include/ +6 more | +627 | 33 replies | |
| 7/10 | Reclaim stacks via a shrinkerinclude/ kernel/ | +357 | 0 replies | |
| 8/10 | Set PF_RECLAIMABLE_STACK in various placesdrivers/ fs/ fs/ +5 more | +82 | 2 replies | |
| 9/10 | x86: Enable reclaimable stacksarch/ arch/ | +6 | 0 replies | |
| 10/10 | arm64: Enable reclaimable stacksarch/ arch/ | +6 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [RFC 00/10] Reclaimable kernel stacks
This RFC is a different approach to reducing kernel stack usage from the earlier dynamic kernel stack RFC [1]. This patch series aims to reduce the cost of kernel stacks by partially reclaiming stacks of blocked tasks when it is safe to do so.
On Android, system processes typically have 2000-3000 threads. App processes add 1000s more threads on top of this. The number of app processes varies based on device RAM size, but the end result is that 1-2% of system RAM is consumed by kernel stacks. However, most of these threads spend extended periods of time blocked. As such, reclaiming blocked kernel stacks can reduce total kernel stack memory usage by upwards of 50% in various multi-tasking test cases.
When a task is blocked, we know exactly where the top of its stack is and can reclaim any pages past that point. Since any accesses to that portion of the stack are bugs like use-after-return or buffer overflow, turning those invalid accesses into hard crashes could even be considered a positive.
Tracking blocked state and when it is safe to reclaim a stack is done via a series of hooks in the scheduler. The actual reclaim of stacks is done asynchronously in a shrinker.
Once a task's stack has been reclaimed, it cannot be rescheduled until its stack is repopulated. Although there can be a repopulation fast path within the scheduler, reliably allocating memory to repopulate the stack requires a fallback path that defers the repopulation and wakeup to a workqueue context that can use GFP_KERNEL.
The primary challenge is avoiding reclaim deadlocks. If a task blocks while holding a lock used by direct reclaim and then has its stack reclaimed, using GFP_KERNEL to reallocate its stack risks deadlock. To avoid this, only tasks which are known not to hold any locks upon which reclaim depends are considered eligible for stack reclaim. Automatically inferring this property is not feasible, so instead a new PF_RECLAIMABLE_STACK task flag is used to annotate blocking locations that are known safe. While annotating all safe blocking locations is not feasible, the vast majority of userspace threads block using a fairly small number of syscalls - futex, epoll, nanosleep, etc. The 10 annotations added in this series cover >95% of userspace threads on Android based on my testing. Since missing annotations are leaving an optimization on the table rather than an actual bug, other annotations can be added later as needed.
Although reclaimable stacks will not cause reclaim to deadlock, it does introduce a dependency on needing to allocate memory before an OOM victim can exit, as reclaimed stacks need to be repopulated before their tasks can run. While the OOM reaper will still be able to immediately free the victim's mm, the freeing of non-mm memory may be delayed. This can lead to more OOM kills. While this is not a significant concern on Android due to the reliance on lmkd over the kernel OOM killer, it may be a concern on other systems.
This RFC was developed primarily on 6.18 and 7.1 based kernels. I have done fairly heavy stress testing, but it has not yet been deployed to any production systems. If initial feedback on the RFC is somewhat positive, I will work on deploying it to production systems for further stability and performance testing as well as resolving the handful of TODOs left in the RFC.
[1] https://lore.kernel.org/linux-mm/[email protected]/
David Stevens (10):
Add !MEMCG memcg_list_lru_alloc implementation mm/vmalloc: Skip vmallocinfo NUMA stats for VM_SPARSE
fork: refactor vmap stack alloc/free into helpers
mm: vmalloc: support creating aligned vm areas
fork: allocate reclaimable stacks with VM_SPARSE Reclaim memory from blocked kernel stacks Reclaim stacks via a shrinker
Set PF_RECLAIMABLE_STACK in various places
x86: Enable reclaimable stacks
arm64: Enable reclaimable stacks
arch/Kconfig | 18 + arch/arm64/Kconfig | 1 + arch/arm64/include/asm/processor.h | 5 + arch/x86/Kconfig | 1 + arch/x86/include/asm/processor.h | 5 + drivers/android/binder/thread.rs | 14 + fs/eventpoll.c | 3 + fs/pipe.c | 28 +- fs/select.c | 3 + include/linux/list_lru.h | 7 +- include/linux/sched.h | 44 +- include/linux/sched/task_stack.h | 23 + include/linux/vmalloc.h | 2 + kernel/Makefile | 2 + kernel/fork.c | 140 +++++- kernel/futex/waitwake.c | 3 + kernel/sched/core.c | 18 +- kernel/sched/sched.h | 3 + kernel/signal.c | 36 +- kernel/stack_shrinker.c | 760 +++++++++++++++++++++++++++++ kernel/stack_shrinker.h | 58 +++ kernel/time/hrtimer.c | 3 + mm/vmalloc.c | 27 +- rust/kernel/task.rs | 16 + 24 files changed, 1175 insertions(+), 45 deletions(-) create mode 100644 kernel/stack_shrinker.c create mode 100644 kernel/stack_shrinker.h
base-commit: 8d3ae59288f1e7d58d76558a6ee96d533bc5019f
Signature · 1 line
2.55.0.897.gb25b4bd76c-goog