Dynamic Kernel Stacks

Store the task pointer in the ptes of the unpopulated pages of dynamic stacks, to allow the vm_struct pointer to be retrieved without relying on any locks or current.

Quoted 3 lines · David Stevens

Store the task pointer in the ptes of the unpopulated pages of dynamic stacks, to allow the vm_struct pointer to be retrieved without relying on any locks or current.

You fail to explain why you can't use current. Changelogs have to describe the WHY and not the WHAT.

Add missing ENCODE_FRAME_POINTER macro invocation into FRED_ENTER macro, to prevent the unwinder from encountering a NULL stack frame pointer when CONFIG_UNWINDER_FRAME_POINTER is enabled

Fixes 14619d912b65 ("x86/fred: FRED entry/exit and dispatch code")

Signed-off-by David Stevens <[email protected]>

---
 arch/x86/entry/entry_64_fred.S | 2 ++
 1 file changed, 2 insertions(+)

arch/x86/entry/entry_64_fred.S +2 -1

index 894f7f16eb80..119b8214748e 100644
@@ -7,6 +7,7 @@
 #include <linux/kvm_types.h>
 
 #include <asm/asm.h>
+#include <asm/frame.h>
 #include <asm/fred.h>
 #include <asm/segment.h>
 

@@ -19,6 +20,7 @@
        UNWIND_HINT_END_OF_STACK
        ANNOTATE_NOENDBR
        PUSH_AND_CLEAR_REGS
+       ENCODE_FRAME_POINTER
        movq    %rsp, %rdi      /* %rdi -> pt_regs */
 .endm
 
--

2.54.0.rc2.544.gc7ae2d5bb8-goog

Setting aside the rest of the RFC, should this patch be merged? If someone tries to use CONFIG_UNWINDER_FRAME_POINTER on a system with FRED, they'll get a "kernel stack regs ... has bad 'bp' value" warning in the logs the first time the unwinder runs.

Thanks,
David

Quoted 31 lines · David Stevens

Add missing ENCODE_FRAME_POINTER macro invocation into FRED_ENTER macro, to prevent the unwinder from encountering a NULL stack frame pointer when CONFIG_UNWINDER_FRAME_POINTER is enabled

Fixes 14619d912b65 ("x86/fred: FRED entry/exit and dispatch code")

Signed-off-by David Stevens <[email protected]>

---
 arch/x86/entry/entry_64_fred.S | 2 ++
 1 file changed, 2 insertions(+)

arch/x86/entry/entry_64_fred.S +2 -0

index 894f7f16eb80..119b8214748e 100644
@@ -7,6 +7,7 @@
 #include <linux/kvm_types.h>
 
 #include <asm/asm.h>
+#include <asm/frame.h>
 #include <asm/fred.h>
 #include <asm/segment.h>
 

@@ -19,6 +20,7 @@
        UNWIND_HINT_END_OF_STACK
        ANNOTATE_NOENDBR
        PUSH_AND_CLEAR_REGS
+       ENCODE_FRAME_POINTER
        movq    %rsp, %rdi      /* %rdi -> pt_regs */
 .endm
 

Quoted 3 lines

--
2.54.0.rc2.544.gc7ae2d5bb8-goog

Quoted 40 lines · David Stevens

Setting aside the rest of the RFC, should this patch be merged? If someone tries to use CONFIG_UNWINDER_FRAME_POINTER on a system with FRED, they'll get a "kernel stack regs ... has bad 'bp' value" warning in the logs the first time the unwinder runs.

Thanks,
David

Quoted 31 lines · David Stevens

Add missing ENCODE_FRAME_POINTER macro invocation into FRED_ENTER macro, to prevent the unwinder from encountering a NULL stack frame pointer when CONFIG_UNWINDER_FRAME_POINTER is enabled

Fixes 14619d912b65 ("x86/fred: FRED entry/exit and dispatch code")

Signed-off-by David Stevens <[email protected]>

---
arch/x86/entry/entry_64_fred.S | 2 ++

1 file changed, 2 insertions(+)

arch/x86/entry/entry_64_fred.S +2 -0

index 894f7f16eb80..119b8214748e 100644
@@ -7,6 +7,7 @@
 #include <linux/kvm_types.h>
 
 #include <asm/asm.h>
+#include <asm/frame.h>
 #include <asm/fred.h>
 #include <asm/segment.h>
 

@@ -19,6 +20,7 @@
       UNWIND_HINT_END_OF_STACK
       ANNOTATE_NOENDBR
       PUSH_AND_CLEAR_REGS
+       ENCODE_FRAME_POINTER
       movq    %rsp, %rdi      /* %rdi -> pt_regs */

.endm

Quoted 4 lines

Quoted 3 lines

--
2.54.0.rc2.544.gc7ae2d5bb8-goog

Yes, it should.

The macroization is weird and I'm somewhat confused about how it is done but I also don't really care because it isn't like frame pointer use is common on x86-64.

arch/x86/mm/fault.c +16 -0

+
+	new_sp = regs->sp - (FRED_CONFIG_REDZONE_AMOUNT << 6);
+	new_sp &= FRED_STACK_FRAME_RSP_MASK;
+	data_len = sizeof(struct fred_frame);
+	new_sp -= data_len;
+
+	memcpy((void *)new_sp, regs, data_len);
+
+	return new_sp;
+}
+
+__visible noinstr unsigned long switch_to_kstack(struct pt_regs *regs)
+{
+	return copy_stack_data(regs);
+}
+

On Fri, Apr 24 2026 at 12:14, David Stevens wrote:

As promised, I've came around to look at this part too.

arch/x86/mm/fault.c handle_page_fault(struct pt_regs *regs, unsigned long error_code, · line 1488

Quoted 6 lines

arch/x86/mm/fault.c +6 -0

+#ifdef CONFIG_DYNAMIC_STACK
+
+static noinstr unsigned long copy_stack_data(struct pt_regs *regs)
+{
+     unsigned long new_sp;
+     unsigned long data_len;

Just a minor issue. Please read and comply with the coding standards documented in https://docs.kernel.org/process/maintainer-tip.html

Quoted 16 lines

arch/x86/mm/fault.c +16 -0

+
+     new_sp = regs->sp - (FRED_CONFIG_REDZONE_AMOUNT << 6);
+     new_sp &= FRED_STACK_FRAME_RSP_MASK;
+     data_len = sizeof(struct fred_frame);
+     new_sp -= data_len;
+
+     memcpy((void *)new_sp, regs, data_len);
+
+     return new_sp;
+}
+
+__visible noinstr unsigned long switch_to_kstack(struct pt_regs *regs)
+{
+     return copy_stack_data(regs);
+}
+

arch/x86/mm/fault.c handle_page_fault(struct pt_regs *regs, unsigned long error_code, · line 1514

Quoted 10 lines

arch/x86/mm/fault.c +10 -0

+#define ALIGN_TO_STACK(addr) ((addr) & ~(THREAD_ALIGN - 1))
+
+__visible noinstr unsigned long handle_dynamic_stack_kernel_faults(struct pt_regs *regs)
+{
+     unsigned long address;
+     struct task_struct *tsk;
+     bool on_stack;
+
+     address = fred_event_data(regs);
+     if (fault_in_kernel_space(address) && !in_nmi()) {

Assume the following scenario:

CPU runs in user space
NMI is raised
NMI runs on SL0
NMI execution triggers the stack guard page

Don't tell me that's unlikely. As discussed before unlikely does not exist. You just have to enable enough debug muck, build with a compiler which aggressively out of lines code (hint CLANG) and then end up in a deep perf/trace/bpf callchain.

arch/x86/mm/fault.c handle_page_fault(struct pt_regs *regs, unsigned long error_code, · line 1527

Quoted 13 lines

arch/x86/mm/fault.c +13 -0

+             tsk = task_from_stack_address(address);
+
+             if (tsk && dynamic_stack_fault(tsk, address, &on_stack)) {
+                     WARN_ON_ONCE(tsk != current &&
+                                  ALIGN_TO_STACK(regs->sp) != ALIGN_TO_STACK(address));
+                     return 0;
+             }
+     }
+
+     /*
+      * The regular fault handler won't sleep when executing in an
+      * atomic context, so we can complete the #PF directly on the
+      * #PF stack.

Q: What guarantees you that in_atomic() is giving you always the right answer?
A: Nothing

Why?

If there is a #PF (not caused by the thread stack guard) on a SL>0 stack _before_ the kernel reached the point where it increments preempt_count, then in_atomic() returns false.

As a consequence you copy stack data around to the same place, which should be benign, but it is well understood that memcpy() source and destination areas _must_ not overlap. That's UB, no?

I know that should not happen, but that doesn't make it less UB :)

arch/x86/mm/fault.c handle_page_fault(struct pt_regs *regs, unsigned long error_code, · line 1532

Quoted 5 lines

arch/x86/mm/fault.c +5 -0

+      */
+     if (in_atomic())
+             return (unsigned long)regs;
+     else
+             return copy_stack_data(regs);

Thanks,

tglx

On Fri, Jun 26, 2026 at 3:39 PM Thomas Gleixner <[email protected]> wrote:
That is definitely a situation that could happen, I hadn't thought of it. Luckily, I don't think we need NMI-safe re-entrancy to handle this, since an NMI on SL0 couldn't have interrupted a dynamic stack fault.

Quoted 5 lines

As a consequence you copy stack data around to the same place, which should be benign, but it is well understood that memcpy() source and destination areas _must_ not overlap. That's UB, no?

I know that should not happen, but that doesn't make it less UB :)

That sort of fault would probably end up either with a page_fault_oops() or risk a context switch with SL>0, but we shouldn't make debugging harder with UB.

Quoted 5 lines · David Stevens

On Fri, Jun 26, 2026 at 3:39 PM Thomas Gleixner <[email protected]> wrote:
That is definitely a situation that could happen, I hadn't thought of it. Luckily, I don't think we need NMI-safe re-entrancy to handle this, since an NMI on SL0 couldn't have interrupted a dynamic stack fault.

Correct.

Quoted 9 lines

Quoted 5 lines

As a consequence you copy stack data around to the same place, which should be benign, but it is well understood that memcpy() source and destination areas _must_ not overlap. That's UB, no?

I know that should not happen, but that doesn't make it less UB :)

That sort of fault would probably end up either with a page_fault_oops() or risk a context switch with SL>0, but we shouldn't make debugging harder with UB.

Definitely not :)

You can avoid quite some of the problems right in the ASM entry code. Instead of checking event type/vector and then going to the stack switch logic, you can check the stack level on which the event happened first. If it's not 0, then there is nothing to do.

Quoted 6 lines · Thomas Gleixner

Quoted 3 lines

That sort of fault would probably end up either with a page_fault_oops() or risk a context switch with SL>0, but we shouldn't make debugging harder with UB.

Definitely not :)

Quoted 6 lines

You can avoid quite some of the problems right in the ASM entry code. Instead of checking event type/vector and then going to the stack switch logic, you can check the stack level on which the event happened first. If it's not 0, then there is nothing to do.

To be fair, for this *specific* case, memmove() does exist for a reason :)

Quoted 14 lines · H. Peter Anvin

Quoted 12 lines · Thomas Gleixner

Quoted 3 lines

That sort of fault would probably end up either with a page_fault_oops() or risk a context switch with SL>0, but we shouldn't make debugging harder with UB.

Definitely not :)

You can avoid quite some of the problems right in the ASM entry code. Instead of checking event type/vector and then going to the stack switch logic, you can check the stack level on which the event happened first. If it's not 0, then there is nothing to do.

Quoted 1 line

To be fair, for this *specific* case, memmove() does exist for a reason :)

It does, but why bother doing actual work, if you know upfront that it's not required :)

Quoted 4 lines · Thomas Gleixner

Quoted 2 lines · H. Peter Anvin

To be fair, for this *specific* case, memmove() does exist for a reason :)

Quoted 2 lines

It does, but why bother doing actual work, if you know upfront that it's not required :)

Of course.

Then there is the very open question if this thing has legs at all.

I'm still interested in using SL1 for the interrupt stack as I think that could be a performance boost.

The question is then: is this approach something that is fundamentally untenable in the kernel

Quoted 2 lines · David Stevens

The question is then: is this approach something that is fundamentally untenable in the kernel

Yes. Fundamentally untenable.

Not allowing stack faults has been a wonderful simplification. It's one of those things that just plain makes the kernel easier to maintain. Saving low single digits of system memory is not exactly making me eager to go back to the harder-to-maintain days.

I seriously doubt that this 1% is the lowest hanging fruit for memory bloat on these systems. ;)

Quoted 10 lines · Dave Hansen

Quoted 2 lines · David Stevens

The question is then: is this approach something that is fundamentally untenable in the kernel

Yes. Fundamentally untenable.

Not allowing stack faults has been a wonderful simplification. It's one of those things that just plain makes the kernel easier to maintain. Saving low single digits of system memory is not exactly making me eager to go back to the harder-to-maintain days.

Quoted 2 lines

I seriously doubt that this 1% is the lowest hanging fruit for memory bloat on these systems. 😉

This true until, in a fleet of millions of machines, you encounter a one-in-a-billion chance of a stack overflow. You are then forced to double the statically allocated kernel stacks on every machine, paying a memory tax even though 99.999..% of threads never exceed 4K. This overhead accumulates to petabytes of wasted capacity.

Quoted 11 lines · Pasha Tatashin

Quoted 10 lines · Dave Hansen

Quoted 2 lines · David Stevens

The question is then: is this approach something that is fundamentally untenable in the kernel

Yes. Fundamentally untenable.

Not allowing stack faults has been a wonderful simplification. It's one of those things that just plain makes the kernel easier to maintain. Saving low single digits of system memory is not exactly making me eager to go back to the harder-to-maintain days.

Quoted 7 lines

Quoted 2 lines

I seriously doubt that this 1% is the lowest hanging fruit for memory bloat on these systems. 😉

This true until, in a fleet of millions of machines, you encounter a one-in-a-billion chance of a stack overflow. You are then forced to double the statically allocated kernel stacks on every machine, paying a memory tax even though 99.999..% of threads never exceed 4K. This overhead accumulates to petabytes of wasted capacity.

I don't disagree with you. But, at that point, you're picking your
poison: bugs dynamic kernel stacks versus crashes from stack overflows.

At some point, I might be able to be talked into dynamic stack as a FRED-only feature. But FRED isn't widespread enough to go to the trouble today. I'm sure the folks who want this also don't want to wait until all the devices in the field have FRED because that even *longer* off.

Quoted 11 lines · Dave Hansen

Quoted 2 lines · David Stevens

The question is then: is this approach something that is fundamentally untenable in the kernel

Yes. Fundamentally untenable.

Not allowing stack faults has been a wonderful simplification. It's one of those things that just plain makes the kernel easier to maintain. Saving low single digits of system memory is not exactly making me eager to go back to the harder-to-maintain days.

Quoted 2 lines

I seriously doubt that this 1% is the lowest hanging fruit for memory bloat on these systems. ;)

This true until, in a fleet of millions of machines, you encounter a one-in-a-billion chance of a stack overflow. You are then forced to double the statically allocated kernel stacks on every machine, paying a memory tax even though 99.999..% of threads never exceed 4K. This overhead accumulates to petabytes of wasted capacity.

Pasha

Quoted 15 lines · On Fri, 24 Apr 2026 21:35:20 +0000 Pasha Tatashin <[email protected]>…

Quoted 13 lines · Dave Hansen

Quoted 2 lines · David Stevens

The question is then: is this approach something that is fundamentally untenable in the kernel

Yes. Fundamentally untenable.

Not allowing stack faults has been a wonderful simplification. It's one of those things that just plain makes the kernel easier to maintain. Saving low single digits of system memory is not exactly making me eager to go back to the harder-to-maintain days.

I seriously doubt that this 1% is the lowest hanging fruit for memory bloat on these systems. ;)

Quoted 5 lines

This true until, in a fleet of millions of machines, you encounter a one-in-a-billion chance of a stack overflow. You are then forced to double the statically allocated kernel stacks on every machine, paying a memory tax even though 99.999..% of threads never exceed 4K. This overhead accumulates to petabytes of wasted capacity.

And then you hit a stack fault in some path where you can't sleep and there isn't any available kernel memory.

An alternative idea is to arrange for some system calls to sleep in userspace, so when the thread is woken it re-executes the system call. It then makes sense to assign the kernel stack to the process when it enters the kernel.
That might mean that you don't need a kernel stack for all the threads sleeping in futex() - it might even be possible to do the retry in userspace saving the second kernel entry most of the time. It is all 'hard and difficult' though.

The easier solution is to rewrite the system code so it doesn't have 1000s of threads :-)

Quoted 5 lines

This true until, in a fleet of millions of machines, you encounter a one-in-a-billion chance of a stack overflow. You are then forced to double the statically allocated kernel stacks on every machine, paying a memory tax even though 99.999..% of threads never exceed 4K. This overhead accumulates to petabytes of wasted capacity.

And then you hit a stack fault in some path where you can't sleep and there isn't any available kernel memory.

An alternative idea is to arrange for some system calls to sleep in userspace, so when the thread is woken it re-executes the system call. It then makes sense to assign the kernel stack to the process when it enters the kernel.

Quoted 5 lines

Quoted 5 lines

This true until, in a fleet of millions of machines, you encounter a one-in-a-billion chance of a stack overflow. You are then forced to double the statically allocated kernel stacks on every machine, paying a memory tax even though 99.999..% of threads never exceed 4K. This overhead accumulates to petabytes of wasted capacity.

Quoted 7 lines

And then you hit a stack fault in some path where you can't sleep and there isn't any available kernel memory.

An alternative idea is to arrange for some system calls to sleep in userspace, so when the thread is woken it re-executes the system call. It then makes sense to assign the kernel stack to the process when it enters the kernel.

There are probably other ways to do this without handling exceptions.

For instance, let's say you always *map* 16k of stack for each process. But, after context switching out, you take a look at 4x8b pte_t's that were mapping the kernel stack. If the _PAGE_ACCESSED bit is clear, you can just clear _PAGE_PRESENT and reclaim the page.

If you don't want the overhead in the normal context switch path, you reclaim in a shrinker, at the cost of needing locking to coordinate with the scheduler.

A simple rule would be: a thread that ever accesses a page gets to keep it forever. They're never reclaimed after being accessed, only before.

For that, the worst case is that you go to schedule a new thread and can't allocate memory fill in the 4 pte_t's. You can't run it until you or some other CPU goes and does some reclaim.

Needing memory in the middle of schedule() is generally a no-go. But its a lot better than not being able to continue _execution_ of a kernel thread at *ALL*, possibly in a non-preemptible context, like when you do it in a #PF.

Basically, I think there's a way to do this that limits the kernel blast radius to _mostly_ being a core mm problem.

What else has been considered before the #PF-based mechanism?

Quoted 5 lines · David Laight

Quoted 5 lines

This true until, in a fleet of millions of machines, you encounter a one-in-a-billion chance of a stack overflow. You are then forced to double the statically allocated kernel stacks on every machine, paying a memory tax even though 99.999..% of threads never exceed 4K. This overhead accumulates to petabytes of wasted capacity.

Quoted 7 lines

And then you hit a stack fault in some path where you can't sleep and there isn't any available kernel memory.

An alternative idea is to arrange for some system calls to sleep in userspace, so when the thread is woken it re-executes the system call. It then makes sense to assign the kernel stack to the process when it enters the kernel.

There are probably other ways to do this without handling exceptions.

For instance, let's say you always *map* 16k of stack for each process. But, after context switching out, you take a look at 4x8b pte_t's that were mapping the kernel stack. If the _PAGE_ACCESSED bit is clear, you can just clear _PAGE_PRESENT and reclaim the page.

If you don't want the overhead in the normal context switch path, you reclaim in a shrinker, at the cost of needing locking to coordinate with the scheduler.

A simple rule would be: a thread that ever accesses a page gets to keep it forever. They're never reclaimed after being accessed, only before.

For that, the worst case is that you go to schedule a new thread and can't allocate memory fill in the 4 pte_t's. You can't run it until you or some other CPU goes and does some reclaim.

Needing memory in the middle of schedule() is generally a no-go. But its a lot better than not being able to continue _execution_ of a kernel thread at *ALL*, possibly in a non-preemptible context, like when you do it in a #PF.

Basically, I think there's a way to do this that limits the kernel blast radius to _mostly_ being a core mm problem.

What else has been considered before the #PF-based mechanism?

Quoted 4 lines

Needing memory in the middle of schedule() is generally a no-go. But its a lot better than not being able to continue _execution_ of a kernel thread at *ALL*, possibly in a non-preemptible context, like when you do it in a #PF.

I don't think this is different from the current proposal from a memory allocation standpoint. Both proposals effectively maintain a pool of preallocated pages used to fill the current thread's stack. They vary substantially in when the pages are put into the page tables, but both need to allocate during schedule().

Quoted 2 lines · David Stevens

The question is then: is this approach something that is fundamentally untenable in the kernel

Yes. Fundamentally untenable.

Not allowing stack faults has been a wonderful simplification. It's one of those things that just plain makes the kernel easier to maintain. Saving low single digits of system memory is not exactly making me eager to go back to the harder-to-maintain days.

I seriously doubt that this 1% is the lowest hanging fruit for memory bloat on these systems. ;)

It is worth noting that this was one of the VERY early design decisions that has shaped Linux from the beginning:

- No swapping of kernel memory
- Kernel stacks are statically allocated

Quoted 5 lines · H. Peter Anvin

It is worth noting that this was one of the VERY early design decisions that has shaped Linux from the beginning:

- No swapping of kernel memory
- Kernel stacks are statically allocated

...

One other bit to add here: In the past, kernel faults on kernel memory have been allowed, like to populate vmalloc() page table entries into the parts of the page tables that are not shared across processes. Even *that* turned out to be too much of a pain even though it didn't involve allocation, and the kernel has been moving away from that.

Overall, are there any particular painpoints you'd like to see flushed out, first?

How would you like to proceed? Would explicitly marking this as an experimental config, in the interim, be more attractive?

Quoted 2 lines · Zach O'Keefe

Overall, are there any particular painpoints you'd like to see flushed out, first?

Handing exceptions in the kernel is hard. Period. That's the pain point. Just look at NMIs, #VC, #MC and the rest of that mess. Just look at how we've moved away from ever taking random page faults in the kernel. Or, heck, randomly taking faults at *all*. We've concentrated them in very specific places, not in general code.

Now you're arguing that the kernel can pretty much take a fault *AND* allocate memory reliably at any point*.

I just don't see the collateral in this series to justify that claim.

The NMI entry code is a disaster because NMIs can happen anywhere. The #VC code is a disaster because #VCs can happen anywhere. Once #PF can happen anywhere*, why won't #PF become a disaster?
[...]
* #PF on stack accesses isn't *quite* as bad as NMI or #VC, I'll give you that. But it's still pretty darn bad.

It would be a completely different story if there was a track record of finding and fixing bugs in the x86 entry code from the authors of this series. But I don't think I've ever seen a single email from your folks before this, much less a review tag or a patch. I'd be much happier if you got Andy L's blessing on this, for example.

Quoted 2 lines

How would you like to proceed? Would explicitly marking this as an experimental config, in the interim, be more attractive?

No.

The enemy here is complexity. *Maintenance* complexity. Being able to compile out some of the complexity helps with debugging. But it doesn't help maintaining the code.

Quoted 10 lines · Dave Hansen

Quoted 2 lines · Zach O'Keefe

Overall, are there any particular painpoints you'd like to see flushed out, first?

Handing exceptions in the kernel is hard. Period. That's the pain point. Just look at NMIs, #VC, #MC and the rest of that mess. Just look at how we've moved away from ever taking random page faults in the kernel. Or, heck, randomly taking faults at *all*. We've concentrated them in very specific places, not in general code.

Quoted 5 lines

Now you're arguing that the kernel can pretty much take a fault *AND* allocate memory reliably at any point*.

I just don't see the collateral in this series to justify that claim.

That is most definitely the zeroth-order thing. Extraordinary claims require extraordinary evidence, and this is certainly an extraordinary claim.

In addition to the *massive* maintainability issue, you also have to consider the additional overheads you will now have to deal with in order to avoid deadlocks.

Almost every OS that have attempted to swap out kernel stacks have been known to suffer from deadlocks under very high memory load.

Quoted 6 lines

The NMI entry code is a disaster because NMIs can happen anywhere. The #VC code is a disaster because #VCs can happen anywhere. Once #PF can happen anywhere*, why won't #PF become a disaster?
[...]
* #PF on stack accesses isn't *quite* as bad as NMI or #VC, I'll give you that. But it's still pretty darn bad.

In some ways, they are actually *worse*.

#PFs need to be able to sleep, because the common case for a #PF in the kernel is that it touched user space. This means #PF needs to be using IST/SL 0. However, this is obviously incompatible with handling #PFs on the kernel stack itself, so now it needs a stack switch. In the common case, it will then need to demote the #PF back onto the normal execution stack, which is complex in its own right.

Now, if you are on a pre-FRED system, the IST entries don't nest, so you absolutely have to make sure you can't get there again through any means whatsoever. With FRED, it isn't quite so dire, but it will still give you lots of fun if that interrupt is one which would like to be demoted off the IRQ stack.

Quoted 6 lines

It would be a completely different story if there was a track record of finding and fixing bugs in the x86 entry code from the authors of this series. But I don't think I've ever seen a single email from your folks before this, much less a review tag or a patch. I'd be much happier if you got Andy L's blessing on this, for example.

Quoted 7 lines

Quoted 2 lines

How would you like to proceed? Would explicitly marking this as an experimental config, in the interim, be more attractive?

No.

The enemy here is complexity. *Maintenance* complexity. Being able to compile out some of the complexity helps with debugging. But it doesn't help maintaining the code.

Indeed. Paravirtualization is a great example of how this works. The PV hooks in the kernel are still a maintenance nightmare 20 years after they were introduced, and mostly that cost is not borne by the people who introduced and benefited from them.

-hpa

Quoted 3 lines

That is most definitely the zeroth-order thing. Extraordinary claims require extraordinary evidence, and this is certainly an extraordinary claim.

I do acknowledge that there is currently a lack of evidence - this is an RFC after all. The question is whether it is possible in principle to produce sufficient evidence. From the Android side of Google, we are willing to carry the RFC patches downstream for a while to build a case for merging them upstream. However, there needs to be at least a possibility of success before we undertake that work. If upstream's position is that dynamic stacks are no good, full stop, and will absolutely never happen, then there's no point in us trying to pursue this avenue further. And I assume those from the datacenter side of the company are in a similar position.

Quoted 2 lines · Zach O'Keefe

Overall, are there any particular painpoints you'd like to see flushed out, first?

Handing exceptions in the kernel is hard. Period. That's the pain point. Just look at NMIs, #VC, #MC and the rest of that mess. Just look at how we've moved away from ever taking random page faults in the kernel. Or, heck, randomly taking faults at *all*. We've concentrated them in very specific places, not in general code.

Now you're arguing that the kernel can pretty much take a fault *AND* allocate memory reliably at any point*.

I just don't see the collateral in this series to justify that claim.

The NMI entry code is a disaster because NMIs can happen anywhere. The #VC code is a disaster because #VCs can happen anywhere. Once #PF can happen anywhere*, why won't #PF become a disaster?

It would be a completely different story if there was a track record of finding and fixing bugs in the x86 entry code from the authors of this series. But I don't think I've ever seen a single email from your folks before this, much less a review tag or a patch. I'd be much happier if you got Andy L's blessing on this, for example.

Quoted 2 lines

How would you like to proceed? Would explicitly marking this as an experimental config, in the interim, be more attractive?

No.

The enemy here is complexity. *Maintenance* complexity. Being able to compile out some of the complexity helps with debugging. But it doesn't help maintaining the code.

Aside of that the part which worries me most is the IDT hackery. That's fragile as hell and full of unvalidated assumptions. Reading "should not happen" several times in a changelog doesn't make me more confident.

"It is possible for #MCE to occur on the #PF IST stack, but the #MCE
handler shouldn't generate new #PFs. The reentrancy check on the #PF stack will trigger if any recoverable #MCEs do generate #PFs - if there are actually reports of it happening, we can address it then."

Seriously?

We don't wait until the report comes in because the report won't even happen in the worst case:

#PF on IST
...
cmp 0, reentrance
jne abort

#MC
...
#PF rewinds #PF IST
cmp 0, reentrance
jne abort <- Not taken because #MC happened before
it could be set.

IST is fundamentally not suitable for this and I'm sure there are more holes in this.

I haven't looked at the FRED side of affairs yet in detail, but the handwavy explanation about external interrupts having to be moved to stack level 1 and unconditionally bounced back does not really make it appealing. I agree that chapter 8.3.4 in the SDM volume 3 is not really helpful, but papering over the problem without understanding the root cause is not cutting it. If it's a genuine FRED hardware issue, then this needs to be understood and documented.

The x86 folks have spent a lot of time to make the horrific x86 interrupt and exception handling solid and therefore have zero interest to deal with the fallout of something based on "shouldn't happen" assumptions. Either it can prove correctness under all circumstances or not.

I understand the save tons of memory accross a fleet argument, but a large fleet is also a guarantee to trigger all the "should not happen and impropable" issues which are gracefully handwaved away. That's a truly bad tradeoff as it ends up in non-decodable bug reports. What's worse the have to be handled by the maintainers and not necessarily by those who implemented it.

While it seems common opinion that the IST-based solution is fragile, what of FRED? It seems like this is exactly the kind of support needed to avoid some of the aforementioned sw "mess" in various x86 exception handling paths. I agree that it's less-than-ideal that we are forced to downgrade exception levels in the common #PF case, but is that an unsurmountable problem? Pardon my ignorance.

Lastly, I just want to clarify what folks have meant by "extraordinary claims" or "evidence". Aside from the above discussion on FRED exception handling, the "only" other part of this is the allocation.

Are people concerned about memory unavailability, deadlocking-type issues, or something else? We have considerable design freedom here to avoid certain classes of unreliability, but—barring any clever tricks—I don't know if the allocation can be guaranteed to succeed in all conceivable circumstances. I want to ensure that reality does not present a hard blocker.

Zach!

Quoted 6 lines · Zach O'Keefe

While it seems common opinion that the IST-based solution is fragile, what of FRED? It seems like this is exactly the kind of support needed to avoid some of the aforementioned sw "mess" in various x86 exception handling paths. I agree that it's less-than-ideal that we are forced to downgrade exception levels in the common #PF case, but is that an unsurmountable problem? Pardon my ignorance.

The #PF path is considered perfomance critical. But how much the downgrade matters needs actual numbers to analyze under various workload scenarios.

I've not seen numbers to that effect anywhere. The only numbers provided are marketing material about the memory savings on a freshly booted idle machine. There are _zero_ numbers about the actual real world savings, but claims about the PETABYTE savings possible.

Seriously?

Quoted 3 lines

Lastly, I just want to clarify what folks have meant by "extraordinary claims" or "evidence". Aside from the above discussion on FRED exception handling, the "only" other part of this is the allocation.

Clearly anything which is explained with "shouldn't happen" and "unlikely". At cloud scale nothing is unlikely anymore. That's simply the reality of statistical math.

As I pointed out before the same applies to the unexplained upgrade/downgrade game with external interrupts. Such issues cannot be papered over without understanding the root cause as from decades long experience they come inevitably back some time down the road. Cloud scale even guarantees that.

Quoted 6 lines

Are people concerned about memory unavailability, deadlocking-type issues, or something else? We have considerable design freedom here to avoid certain classes of unreliability, but—barring any clever tricks—I don't know if the allocation can be guaranteed to succeed in all conceivable circumstances. I want to ensure that reality does not present a hard blocker.

First of all the failure scenario has to be clearly defined.

Right now, if I'm reading the patches correctly this simply can end up killing the wrong tasks/processes just because an OOM situation results in a depletion of the per CPU cache and the very wrong task which runs into the deep call stack situation ends up in the creek without a paddle.

Given that you even fail to abort a CPU bringup when the allocation of the per CPU stack page cache fails, makes it pretty clear that there has been spent exactly zero thoughts about this problem.

Why the heck does this cache refill call have to be unconditionally in __schedule() where preemption is disabled and therefore GFP_ATOMIC is mandatory? I know "Works for me" (most of the time).
And just because I was looking at the patch in question I found this other insanity:

Review

Quoted 8 lines

(quoted diff) +8 -0

+     /*
+      * Most likely we faulted in the page right next to the last mapped
+      * page in the stack, however, it is possible (but very unlikely) that
+      * the faulted page is actually skips some pages in the stack. Make sure
+      * we do not create  more than one holes in the stack, and map every
+      * page between the current fault  address and the last page that is
+      * mapped in the stack.
+      */

Can anyone with a sane mind and the most minimal understanding of the kernel's inner working explain to me how the kernel can skip "some pages" on the stack?

If the kernel skips a whole page or more then there is a serious bug somewhere. I might be missing something, but again the "very unlikely" wording which handwaves about it is just disgustingly useless.

I disagree with Dave on the RFC status of this series. It's not even close to RFC, it's at PoC status.

Thanks,

tglx

Quoted 3 lines · Thomas Gleixner

If the kernel skips a whole page or more then there is a serious bug somewhere. I might be missing something, but again the "very unlikely" wording which handwaves about it is just disgustingly useless.

FRAME_WARN accepts values up to 8192 bytes, and it can always be ignored or simply disabled. If a stack frame is larger than 4k, then

it's entirely possible for the code and compiler to align in a way where the first access in the frame skips a page in the stack. I think we agree that such code would be highly suspect and (hopefully) would only exist in out-of-tree drivers.

But it's something the kernel build system accepts today. Dynamic kernel stacks suddenly turning that into a runtime kernel panic seems like exactly the sort of edge case that we would get yelled at for not addressing.

Zach!

Quoted 6 lines · Zach O'Keefe

While it seems common opinion that the IST-based solution is fragile, what of FRED? It seems like this is exactly the kind of support needed to avoid some of the aforementioned sw "mess" in various x86 exception handling paths. I agree that it's less-than-ideal that we are forced to downgrade exception levels in the common #PF case, but is that an unsurmountable problem? Pardon my ignorance.

The #PF path is considered perfomance critical. But how much the downgrade matters needs actual numbers to analyze under various workload scenarios.

I've not seen numbers to that effect anywhere. The only numbers provided are marketing material about the memory savings on a freshly booted idle machine. There are _zero_ numbers about the actual real world savings, but claims about the PETABYTE savings possible.

Seriously?

Quoted 3 lines

Lastly, I just want to clarify what folks have meant by "extraordinary claims" or "evidence". Aside from the above discussion on FRED exception handling, the "only" other part of this is the allocation.

Clearly anything which is explained with "shouldn't happen" and "unlikely". At cloud scale nothing is unlikely anymore. That's simply the reality of statistical math.

As I pointed out before the same applies to the unexplained upgrade/downgrade game with external interrupts. Such issues cannot be papered over without understanding the root cause as from decades long experience they come inevitably back some time down the road. Cloud scale even guarantees that.

Quoted 6 lines

Are people concerned about memory unavailability, deadlocking-type issues, or something else? We have considerable design freedom here to avoid certain classes of unreliability, but—barring any clever tricks—I don't know if the allocation can be guaranteed to succeed in all conceivable circumstances. I want to ensure that reality does not present a hard blocker.

First of all the failure scenario has to be clearly defined.

Right now, if I'm reading the patches correctly this simply can end up killing the wrong tasks/processes just because an OOM situation results in a depletion of the per CPU cache and the very wrong task which runs into the deep call stack situation ends up in the creek without a paddle.

Given that you even fail to abort a CPU bringup when the allocation of the per CPU stack page cache fails, makes it pretty clear that there has been spent exactly zero thoughts about this problem.

Why the heck does this cache refill call have to be unconditionally in __schedule() where preemption is disabled and therefore GFP_ATOMIC is mandatory? I know "Works for me" (most of the time).

And just because I was looking at the patch in question I found this other insanity:

Review

Quoted 8 lines

(quoted diff) +8 -0

+     /*
+      * Most likely we faulted in the page right next to the last mapped
+      * page in the stack, however, it is possible (but very unlikely) that
+      * the faulted page is actually skips some pages in the stack. Make sure
+      * we do not create  more than one holes in the stack, and map every
+      * page between the current fault  address and the last page that is
+      * mapped in the stack.
+      */

Can anyone with a sane mind and the most minimal understanding of the kernel's inner working explain to me how the kernel can skip "some pages" on the stack?

If the kernel skips a whole page or more then there is a serious bug somewhere. I might be missing something, but again the "very unlikely" wording which handwaves about it is just disgustingly useless.

I disagree with Dave on the RFC status of this series. It's not even close to RFC, it's at PoC status.

Thanks,

tglx

On Fri, Jun 19, 2026 at 2:59 PM Thomas Gleixner <[email protected]> wrote:

Thomas, thanks again for taking the time to look into this and help out.

Quoted 10 lines

Zach!

Quoted 6 lines · Zach O'Keefe

While it seems common opinion that the IST-based solution is fragile, what of FRED? It seems like this is exactly the kind of support needed to avoid some of the aforementioned sw "mess" in various x86 exception handling paths. I agree that it's less-than-ideal that we are forced to downgrade exception levels in the common #PF case, but is that an unsurmountable problem? Pardon my ignorance.

Quoted 3 lines

The #PF path is considered perfomance critical. But how much the downgrade matters needs actual numbers to analyze under various workload scenarios.

Ya, that's my concern as well, as I don't have a good intuition for how perf critical kernel #PF is for real workloads. If this is your primary concern, I'll take that as a _good_ thing ; i.e. there's nothing architecturally stopping us from doing this downgrade safely. We'll still need the analysis, but that can be a later stage -- we're more than happy to get this data for all.

Quoted 6 lines

I've not seen numbers to that effect anywhere. The only numbers provided are marketing material about the memory savings on a freshly booted idle machine. There are _zero_ numbers about the actual real world savings, but claims about the PETABYTE savings possible.

Seriously?

This is actually the most understood aspect. With O(100B) active tasks fleetwide at any point, it only takes an average savings of O(10KiB) per task to get to 1PiB. At least for our fleet, we know the % of tasks that use only 4KiB, 8KiB, or require the full 16KiB, and the math confirms that we expect O(PiB) aggregate savings. The % of stacks requiring the full 16KiB is minuscule, but it still occurs at a rate higher than what we can tolerate for SO panics. Given the vast majority of stacks never exceed the first 4KiB, this enables the significant opportunity.

Quoted 52 lines

Quoted 3 lines

Lastly, I just want to clarify what folks have meant by "extraordinary claims" or "evidence". Aside from the above discussion on FRED exception handling, the "only" other part of this is the allocation.

Clearly anything which is explained with "shouldn't happen" and "unlikely". At cloud scale nothing is unlikely anymore. That's simply the reality of statistical math.

As I pointed out before the same applies to the unexplained upgrade/downgrade game with external interrupts. Such issues cannot be papered over without understanding the root cause as from decades long experience they come inevitably back some time down the road. Cloud scale even guarantees that.

Quoted 6 lines

Are people concerned about memory unavailability, deadlocking-type issues, or something else? We have considerable design freedom here to avoid certain classes of unreliability, but—barring any clever tricks—I don't know if the allocation can be guaranteed to succeed in all conceivable circumstances. I want to ensure that reality does not present a hard blocker.

First of all the failure scenario has to be clearly defined.

Right now, if I'm reading the patches correctly this simply can end up killing the wrong tasks/processes just because an OOM situation results in a depletion of the per CPU cache and the very wrong task which runs into the deep call stack situation ends up in the creek without a paddle.

Given that you even fail to abort a CPU bringup when the allocation of the per CPU stack page cache fails, makes it pretty clear that there has been spent exactly zero thoughts about this problem.

Why the heck does this cache refill call have to be unconditionally in __schedule() where preemption is disabled and therefore GFP_ATOMIC is mandatory? I know "Works for me" (most of the time).

And just because I was looking at the patch in question I found this other insanity:

Review

Quoted 8 lines

(quoted diff) +8 -0

+     /*
+      * Most likely we faulted in the page right next to the last mapped
+      * page in the stack, however, it is possible (but very unlikely) that
+      * the faulted page is actually skips some pages in the stack. Make sure
+      * we do not create  more than one holes in the stack, and map every
+      * page between the current fault  address and the last page that is
+      * mapped in the stack.
+      */

Can anyone with a sane mind and the most minimal understanding of the kernel's inner working explain to me how the kernel can skip "some pages" on the stack?

Quoted 6 lines

If the kernel skips a whole page or more then there is a serious bug somewhere. I might be missing something, but again the "very unlikely" wording which handwaves about it is just disgustingly useless.

I disagree with Dave on the RFC status of this series. It's not even close to RFC, it's at PoC status.

Absolutely understood. I'm more interested in constructively working together (as we can see, we'll need your help) to figure out how the x86 experts want to approach this vs discussing _this_ series. Perhaps it was my mistake to necro this thread instead of starting a new, general discussion. Appologies.

To that end, how would you like to proceed? You may understand the x86 complexities better than anyone, so hopefully you can guide this in the right direction. How would you like us to approach this?

Thanks again for your time, help, and support,
Zach

Quoted 10 lines

Thanks,

tglx

1 PiB for a fleet only makes sense in the context of the size of that fleet.

But it's more than that.

You WILL slow down the general case by this stuff, and so how much actual gain does this imply? What is the mark needed to even get to a break-even point?

To be honest, this and the multikernel proposal are the worst motivated massive changes for no demonstrated value I have seen in a very, very long time.

Quoted 3 lines · Thomas Gleixner

The #PF path is considered perfomance critical. But how much the downgrade matters needs actual numbers to analyze under various workload scenarios.

Ya, that's my concern as well, as I don't have a good intuition for how perf critical kernel #PF is for real workloads. If this is your primary concern, I'll take that as a _good_ thing ; i.e. there's nothing architecturally stopping us from doing this downgrade safely. We'll still need the analysis, but that can be a later stage -- we're more than happy to get this data for all.

Quoted 6 lines

I've not seen numbers to that effect anywhere. The only numbers provided are marketing material about the memory savings on a freshly booted idle machine. There are _zero_ numbers about the actual real world savings, but claims about the PETABYTE savings possible.

Seriously?

This is actually the most understood aspect. With O(100B) active tasks fleetwide at any point, it only takes an average savings of O(10KiB) per task to get to 1PiB. At least for our fleet, we know the % of tasks that use only 4KiB, 8KiB, or require the full 16KiB, and the math confirms that we expect O(PiB) aggregate savings. The % of stacks requiring the full 16KiB is minuscule, but it still occurs at a rate higher than what we can tolerate for SO panics. Given the vast majority of stacks never exceed the first 4KiB, this enables the significant opportunity.

Quoted 11 lines · Thomas Gleixner

Quoted 3 lines

The #PF path is considered perfomance critical. But how much the downgrade matters needs actual numbers to analyze under various workload scenarios.

Ya, that's my concern as well, as I don't have a good intuition for how perf critical kernel #PF is for real workloads. If this is your primary concern, I'll take that as a _good_ thing ; i.e. there's nothing architecturally stopping us from doing this downgrade safely. We'll still need the analysis, but that can be a later stage -- we're more than happy to get this data for all.

No. That's not a later stage optional requirement.

You have a PoC which works for you otherwise you wouldn't have posted it. So you can trivially microbenchmark the costs of the up/downgrade. And that's critical information for us but also for you. If the costs are significant then you really have to think about the tradeoffs.

Care to read Documentation/process/* carefully? It applies to you as it applies to anyone else.

Quoted 7 lines

Quoted 6 lines

I've not seen numbers to that effect anywhere. The only numbers provided are marketing material about the memory savings on a freshly booted idle machine. There are _zero_ numbers about the actual real world savings, but claims about the PETABYTE savings possible.

Seriously?

Quoted 9 lines

This is actually the most understood aspect. With O(100B) active tasks fleetwide at any point, it only takes an average savings of O(10KiB) per task to get to 1PiB. At least for our fleet, we know the % of tasks that use only 4KiB, 8KiB, or require the full 16KiB, and the math confirms that we expect O(PiB) aggregate savings. The % of stacks requiring the full 16KiB is minuscule, but it still occurs at a rate higher than what we can tolerate for SO panics. Given the vast majority of stacks never exceed the first 4KiB, this enables the significant opportunity.

I know that the potential savings are well understood and my understanding of math is sufficient to calculate how much tasks and average saving it takes to save 1PiB on a fleet.

That's a no-brainer, but this is an aggregate saving, which sounds WOW but does not tell much about anything else.

1) What's the actual percentage of savings in relation to the overall memory?

2) Does the saving allow you to get more stuff done on a machine, pack more threads on it?

3) Can you actually downsize the memory on the machines?

4) What is the performance tradeoff for that?

IOW, you fail to tell what the actual benefit of such an intrusive change is. Just boasting an aggregate Petabyte number does not tell anything at all.

Let me give you a trivial example with a scenario which I have access to:

    256  CPUs
    256  GiB Memory
    64k  Threads

Let's assume the full saving of 12k per thread. That sums up to

      64k * 12k = 768MB of memory

which is 0.29% of the total 256 GiB of memory. Not so impressive as the petabyte aggregate number, right?

The workload consumes about 80% of the overall memory and is already constraint on close to 100% CPU utilization.

Now let's assume that the runtime overhead of this amounts to 1% then this is a net loss.

Let me turn that around and use a made up example assuming the 1Mio threads per compute unit taken from some reply in this thread.

Now the full saving of 12k per thread amounts to:

    1M * 12k = 12G

which is 4.7% of the overall available memory. Agreed that's a substantial number.

That 12G saving does not do anything in terms of hardware downsizing.

The only way that has a benefit is when the system is constraint by overall memory consumption, but has quite some compute capacity left.

IOW, if 1M threads hit the memory limit that means that the savings in kernel stack consumed memory allows you to add about 4% (~40k) more threads. If that ups the CPU utilization accordingly then yes, I can see the benefit. But TBH, if that's the case then you are trying to fix a user space implementation problem in the kernel.

That said you really have to describe the scenarios where there is a benefit and I do not buy this "fleet level" argument at all because there is no single fleet which has a uniform workload distribution.

Aside of that. If your argument holds that there are only a few scenarios which require a deep stack, then we are better off to identify them and fix them up rather than trying to hack around the occacional insanity of deep stack usage by adding complexity for complexity sake.

As you say that you have numbers of your fleet which confirm that the vast majority of the stack depth is below 4k, you can surely figure out the information which call chains are actually exceeding the limit.

I prefer to fix such shitty code and downgrade the stacksize in general instead of papering over the underlying issues which probably have been ignored for years if not decades.

Have you ever thought about that instead of adding complexity with a dubious value?

Thanks,

tglx

On Sat, Jun 20, 2026 at 4:34 PM Thomas Gleixner <[email protected]> wrote:

Thomas, thanks for taking the time, as always, for such a thoughtful response.

Quoted 93 lines · Zach O'Keefe

Quoted 7 lines

Ya, that's my concern as well, as I don't have a good intuition for how perf critical kernel #PF is for real workloads. If this is your primary concern, I'll take that as a _good_ thing ; i.e. there's nothing architecturally stopping us from doing this downgrade safely. We'll still need the analysis, but that can be a later stage -- we're more than happy to get this data for all.

No. That's not a later stage optional requirement.

You have a PoC which works for you otherwise you wouldn't have posted it. So you can trivially microbenchmark the costs of the up/downgrade. And that's critical information for us but also for you. If the costs are significant then you really have to think about the tradeoffs.

Care to read Documentation/process/* carefully? It applies to you as it applies to anyone else.

Quoted 10 lines

This is actually the most understood aspect. With O(100B) active tasks fleetwide at any point, it only takes an average savings of O(10KiB) per task to get to 1PiB. At least for our fleet, we know the % of tasks that use only 4KiB, 8KiB, or require the full 16KiB, and the math confirms that we expect O(PiB) aggregate savings. The % of stacks requiring the full 16KiB is minuscule, but it still occurs at a rate higher than what we can tolerate for SO panics. Given the vast majority of stacks never exceed the first 4KiB, this enables the significant opportunity.

I know that the potential savings are well understood and my understanding of math is sufficient to calculate how much tasks and average saving it takes to save 1PiB on a fleet.

That's a no-brainer, but this is an aggregate saving, which sounds WOW but does not tell much about anything else.

1) What's the actual percentage of savings in relation to the overall memory?

2) Does the saving allow you to get more stuff done on a machine, pack more threads on it?

3) Can you actually downsize the memory on the machines?

4) What is the performance tradeoff for that?

IOW, you fail to tell what the actual benefit of such an intrusive change is. Just boasting an aggregate Petabyte number does not tell anything at all.

Let me give you a trivial example with a scenario which I have access to:

    256  CPUs
    256  GiB Memory
    64k  Threads

Let's assume the full saving of 12k per thread. That sums up to

      64k * 12k = 768MB of memory

which is 0.29% of the total 256 GiB of memory. Not so impressive as the petabyte aggregate number, right?

The workload consumes about 80% of the overall memory and is already constraint on close to 100% CPU utilization.

Now let's assume that the runtime overhead of this amounts to 1% then this is a net loss.

Let me turn that around and use a made up example assuming the 1Mio threads per compute unit taken from some reply in this thread.

Now the full saving of 12k per thread amounts to:

    1M * 12k = 12G

which is 4.7% of the overall available memory. Agreed that's a substantial number.

That 12G saving does not do anything in terms of hardware downsizing.

The only way that has a benefit is when the system is constraint by overall memory consumption, but has quite some compute capacity left.

IOW, if 1M threads hit the memory limit that means that the savings in kernel stack consumed memory allows you to add about 4% (~40k) more threads. If that ups the CPU utilization accordingly then yes, I can see the benefit. But TBH, if that's the case then you are trying to fix a user space implementation problem in the kernel.

Quoted 3 lines

That said you really have to describe the scenarios where there is a benefit and I do not buy this "fleet level" argument at all because there is no single fleet which has a uniform workload distribution.

These are good thoughts, thank you. Perhaps I've been too biased by our particular environment—apologies for that.

We (mostly) punt this problem to cluster-level scheduling, which ironically exploits this non-uniformity of workload dynamics to appropriately bin-pack machines and materialize these small savings.

In the general case, I guess a lot hinges on that overhead cost -- in the best (memory-constrained) case.

Quoted 15 lines

Aside of that. If your argument holds that there are only a few scenarios which require a deep stack, then we are better off to identify them and fix them up rather than trying to hack around the occacional insanity of deep stack usage by adding complexity for complexity sake.

As you say that you have numbers of your fleet which confirm that the vast majority of the stack depth is below 4k, you can surely figure out the information which call chains are actually exceeding the limit.

I prefer to fix such shitty code and downgrade the stacksize in general instead of papering over the underlying issues which probably have been ignored for years if not decades.

Have you ever thought about that instead of adding complexity with a dubious value?

There was some (hallway?) talk at LSF/MM about possibly removing direct reclaim, similar to how other operating systems handle it. Now, I don't know how feasible it is (I guess devil is in the detail ;) ), or any details how that would work, but direct reclaim was repeatedly called out as one of the main reasons we can get huge stacks.

So I guess direct reclaim (incl. compaction) is one of the main problematic pieces. Are we aware of other scenarios where we (easily) trigger consumption of larger stacks?

Wild idea: as a first step to test the waters, use smaller stacks on selected kernel threads and disallow direct reclaim/compaction if the stack for the thread is small?

On Tue, 23 Jun 2026 09:50:00 +0200
"David Hildenbrand (Arm)" <[email protected]> wrote:

...

Quoted 6 lines

There was some (hallway?) talk at LSF/MM about possibly removing direct reclaim, similar to how other operating systems handle it. Now, I don't know how feasible it is (I guess devil is in the detail ;) ), or any details how that would work, but direct reclaim was repeatedly called out as one of the main reasons we can get huge stacks.

Quoted 3 lines

So I guess direct reclaim (incl. compaction) is one of the main problematic pieces. Are we aware of other scenarios where we (easily) trigger consumption of larger stacks?

I suspect some printk in error paths could get problematic. Especially if they involve direct writes to a non-trivial console (does that happen any more??).

How much stack does dump_stack() need?

Quoted 3 lines

Wild idea: as a first step to test the waters, use smaller stacks on selected kernel threads and disallow direct reclaim/compaction if the stack for the thread is small?

Or if there isn't much stack space left?

Isn't reclaim one of the places that is actually likely to exceed the stack limit because it is unusual and won't be exercised at most of the kmalloc() call sites?
One you get to the slow bits (especially ones that do IO) it is unlikely that a context-switch to a special worker thread will make much difference. (You might need to boost the thread priority based on the priority of the waiting process.)

Quoted 3 lines · Thomas Gleixner

That said you really have to describe the scenarios where there is a benefit and I do not buy this "fleet level" argument at all because there is no single fleet which has a uniform workload distribution.

These are good thoughts, thank you. Perhaps I've been too biased by our particular environment—apologies for that.

We (mostly) punt this problem to cluster-level scheduling, which ironically exploits this non-uniformity of workload dynamics to appropriately bin-pack machines and materialize these small savings.

In the general case, I guess a lot hinges on that overhead cost -- in the best (memory-constrained) case.

Quoted 9 lines

Aside of that. If your argument holds that there are only a few scenarios which require a deep stack, then we are better off to identify them and fix them up rather than trying to hack around the occacional insanity of deep stack usage by adding complexity for complexity sake.

As you say that you have numbers of your fleet which confirm that the vast majority of the stack depth is below 4k, you can surely figure out the information which call chains are actually exceeding the limit.

Quoted 6 lines

I prefer to fix such shitty code and downgrade the stacksize in general instead of papering over the underlying issues which probably have been ignored for years if not decades.

Have you ever thought about that instead of adding complexity with a dubious value?

Yeah, and that is certainly an alternative path we can explore. I was _hoping_ to be able to maximize the savings here, via the >90% case where 4KiB is sufficient. If we instead play whack-a-mole with the handful of cases requiring > 8KiB, the best case is we can move back to 8KiB stacks. Ironically, I was thinking going this route would be _more_ of a maintenance burden vs having a generalized solution ;)

So as you already confirmed that the number of offenders is really low, it'd be definitely a worthwhile effort to fix that for everyone and also make the compiler people to get their act together 🙂

Quoted 3 lines · Thomas Gleixner

So as you already confirmed that the number of offenders is really low, it'd be definitely a worthwhile effort to fix that for everyone and also make the compiler people to get their act together 🙂

One other thing comes to mind...

We have debugging options for looking at max stack depth, like CONFIG_DEBUG_STACK_USAGE. But no distros seem to turn that on. There's also some ftrace stuff, but I assume none of that is used at runtime either.

Do we need some max stack depth instrumentation that folks will actually turn on? It would be nice if we spat out warnings when folks went through 12k (or 8k) of stack so there was a chance of closing the loop and getting feedback from them.

It could be done with really little overhead if the vmalloc()'d stacks set Accessed=0 on their PTEs and then checked them near vfree(). The PTEs are already getting touched there, so the cachelines should be hot anyway.

The granularity would only be 4k, but it would be so cheap that we could turn it on universally. It would also be 100% deterministic.

There are, of course, more games that could be played with stack depth checks on normal interrupts or NMIs. But those would be less deterministic.

Thoughts?

Quoted 3 lines · Thomas Gleixner

So as you already confirmed that the number of offenders is really low, it'd be definitely a worthwhile effort to fix that for everyone and also make the compiler people to get their act together 🙂

One other thing comes to mind...

We have debugging options for looking at max stack depth, like CONFIG_DEBUG_STACK_USAGE. But no distros seem to turn that on. There's also some ftrace stuff, but I assume none of that is used at runtime either.

Do we need some max stack depth instrumentation that folks will actually turn on? It would be nice if we spat out warnings when folks went through 12k (or 8k) of stack so there was a chance of closing the loop and getting feedback from them.

It could be done with really little overhead if the vmalloc()'d stacks set Accessed=0 on their PTEs and then checked them near vfree(). The PTEs are already getting touched there, so the cachelines should be hot anyway.

The granularity would only be 4k, but it would be so cheap that we could turn it on universally. It would also be 100% deterministic.

There are, of course, more games that could be played with stack depth checks on normal interrupts or NMIs. But those would be less deterministic.

Thoughts?

But it wouldn't tell you what was the large stack? Or we could try to print it inaccurately, and hope the non-overwritten deep part gives us enough insight?

Quoted 2 lines

There are, of course, more games that could be played with stack depth checks on normal interrupts or NMIs. But those would be less deterministic.>

Yes, we run with CONFIG_DEBUG_STACK_USAGE -- which allows us to put together information on max stack depth usage per host/task.

Thanks All.

Yes, we run with CONFIG_DEBUG_STACK_USAGE -- which allows us to put together information on max stack depth usage per host/task.

I'd like to create an upstream recipe for finding deep stacks statically by combining per-frame sizes from --fstack-usage with callgraphs generated using Dan Carpenter's smatch utility. In particular, smatch's selling point is the ability to track indirect calls (i.e., calls through vm_operations, super_operations, etc). While easy to set up, and working well out-of-the-box, this currently results in many false-positive callstack possibilities, so I'm trying to pare down the graph to get to some useable data. Additionally will need to know what exception-originating stacks can possibly be superimposed on the base callstack, for the given configuration -- for example, today, only a few places we'd expect to see #PF stack. Lastly, I've been analyzing perf-like data to authoritatively construct the call graphs as well -- or at least to use for seeding/filtering the static call graph.

Thanks,
Zach

It could be done with really little overhead if the vmalloc()'d stacks set Accessed=0 on their PTEs and then checked them near vfree(). The PTEs are already getting touched there, so the cachelines should be hot anyway.

The granularity would only be 4k, but it would be so cheap that we could turn it on universally. It would also be 100% deterministic.

There are, of course, more games that could be played with stack depth checks on normal interrupts or NMIs. But those would be less deterministic.

Quoted 8 lines · Dave Hansen

It could be done with really little overhead if the vmalloc()'d stacks set Accessed=0 on their PTEs and then checked them near vfree(). The PTEs are already getting touched there, so the cachelines should be hot anyway.

The granularity would only be 4k, but it would be so cheap that we could turn it on universally. It would also be 100% deterministic.

Quoted 2 lines

There are, of course, more games that could be played with stack depth checks on normal interrupts or NMIs. But those would be less deterministic.

I mean, if we just memset64() the whole stack to 0x535441434b544f50 and then searched the stack for the lowest word that wasn't that value at process exit, we'd get a much more granular idea with vanishingly few false positives.

Quoted 3 lines · Thomas Gleixner

So as you already confirmed that the number of offenders is really low, it'd be definitely a worthwhile effort to fix that for everyone and also make the compiler people to get their act together 🙂

One other thing comes to mind...

We have debugging options for looking at max stack depth, like CONFIG_DEBUG_STACK_USAGE. But no distros seem to turn that on. There's also some ftrace stuff, but I assume none of that is used at runtime either.

Do we need some max stack depth instrumentation that folks will actually turn on? It would be nice if we spat out warnings when folks went through 12k (or 8k) of stack so there was a chance of closing the loop and getting feedback from them.

It could be done with really little overhead if the vmalloc()'d stacks set Accessed=0 on their PTEs and then checked them near vfree(). The PTEs are already getting touched there, so the cachelines should be hot anyway.

The granularity would only be 4k, but it would be so cheap that we could turn it on universally. It would also be 100% deterministic.

There are, of course, more games that could be played with stack depth checks on normal interrupts or NMIs. But those would be less deterministic.

Thoughts?

However, I'd still like to work with you to understand what feasible path forward you see, hoping you can proactively steer us away from some of the bigger headaches.

I think we are fine being forward-looking, and only supporting this for FRED (which is on our doorstep). That said, understanding the issues you foresee with the IST approach

would still be valuable, as
it might save us internal trouble should we choose to carry it temporarily to bridge the gap with FRED.

Quoted 7 lines

However, I'd still like to work with you to understand what feasible path forward you see, hoping you can proactively steer us away from some of the bigger headaches.

I think we are fine being forward-looking, and only supporting this for FRED (which is on our doorstep). That said, understanding the issues you foresee with the IST approach

For IST, read
https://docs.google.com/document/d/1hWejnyDkjRRAW-JEsRjA5c9CKLOPc6VKJQsuvODlQEI/edit?usp=sharing

This was written prior to the first public drafts of FRED and SEE specs, and FRED evolved quite a lot from the first draft to where we are today.

Quoted 3 lines

would still be valuable, as
it might save us internal trouble should we choose to carry it temporarily to bridge the gap with FRED.

A lot has been said about the invasiveness and tradeoffs.  I don't wish to dwell on or rehash that.

There is one critical point which has not been made as far as I can tell:

In x86, if delivering an interrupt (NMI or INTR) causes an exception (any exception, not just pagefault), the original interrupt is lost.

If you are aware of this, then in principle you can look at the LAPIC state to figure out when an INTR was missed.  However there is no ability at all to figure out when an NMI was missed.  Worse, it is both vendor and IDT-vs-FRED specific as to whether NMIs are blocked following a missed NMI, and that too is state you cannot recover.

If you wish to make dynamic stacks works, then you *must* find a non-fault based approach on x86.

~Andrew

Dave,

Necroing this thread, as the potential aggregate savings continue to stand out for us on the datacenter side (whereas David is motivated separately, from the consumer device side).

Quoted 6 lines · Dave Hansen

Quoted 2 lines · David Stevens

The question is then: is this approach something that is fundamentally untenable in the kernel

Yes. Fundamentally untenable.

Quoted 7 lines

Not allowing stack faults has been a wonderful simplification. It's one of those things that just plain makes the kernel easier to maintain. Saving low single digits of system memory is not exactly making me eager to go back to the harder-to-maintain days.

I seriously doubt that this 1% is the lowest hanging fruit for memory bloat on these systems. ;)

It is worth noting that this was one of the VERY early design decisions that has shaped Linux from the beginning:

- No swapping of kernel memory
- Kernel stacks are statically allocated
- Physical RAM is mapped into the kernel at all times
- A "monolithic" kernel using function calls, not message passing
- A kernel interface that closely maps to the low-level application API (e.g. each user space thread is a kernel thread.)
- Kernel ABIs and APIs are subject to evolution; stability is only guaranteed in user space.

Those design decisions are, by and large, what has made Linux Linux: a relatively simple, highly performant, and reliable system.