sched: Further restrict the preemption modes
Subsystems (3)
Peter Zijlstra
maintainerPATCH 1/1 sched: Further restrict the preemption modes
[ with 6.18 being an LTS release, it might be a good time for this ]
The introduction of PREEMPT_LAZY was for multiple reasons:
- PREEMPT_RT suffered from over-scheduling, hurting performance compared to !PREEMPT_RT.
- the introduction of (more) features that rely on preemption; like folio_zero_user() which can do large memset() without preemption checks.
(Xen already had a horrible hack to deal with long running hypercalls)
- the endless and uncontrolled sprinkling of cond_resched() -- mostly cargo cult or in response to poor to replicate workloads.
By moving to a model that is fundamentally preemptable these things become manageable and avoid needing to introduce more horrible hacks.
Since this is a requirement; limit PREEMPT_NONE to architectures that do not support preemption at all. Further limit PREEMPT_VOLUNTARY to those architectures that do not yet have PREEMPT_LAZY support (with the eventual goal to make this the empty set and completely remove voluntary preemption and cond_resched() -- notably VOLUNTARY is already limited to !ARCH_NO_PREEMPT.)
This leaves up-to-date architectures (arm64, loongarch, powerpc, riscv, s390, x86) with only two preemption models: full and lazy (like PREEMPT_RT).
While Lazy has been the recommended setting for a while, not all distributions have managed to make the switch yet. Force things along. Keep the patch minimal in case of hard to address regressions that might pop up.
--- kernel/Kconfig.preempt | 3 +++ kernel/sched/core.c | 2 +- kernel/sched/debug.c | 2 +- 3 files changed, 5 insertions(+), 2 deletions(-)
kernel/Kconfig.preempt +3 -0
@@ -16,11 +16,13 @@ config ARCH_HAS_PREEMPT_LAZY choice prompt "Preemption Model" + default PREEMPT_LAZY if ARCH_HAS_PREEMPT_LAZY default PREEMPT_NONE config PREEMPT_NONE bool "No Forced Preemption (Server)" depends on !PREEMPT_RT + depends on ARCH_NO_PREEMPT select PREEMPT_NONE_BUILD if !PREEMPT_DYNAMIC help This is the traditional Linux preemption model, geared towards
@@ -35,6 +37,7 @@ config PREEMPT_NONE config PREEMPT_VOLUNTARY bool "Voluntary Kernel Preemption (Desktop)" + depends on !ARCH_HAS_PREEMPT_LAZY depends on !ARCH_NO_PREEMPT depends on !PREEMPT_RT select PREEMPT_VOLUNTARY_BUILD if !PREEMPT_DYNAMIC
kernel/sched/core.c +1 -1
@@ -7553,7 +7553,7 @@ int preempt_dynamic_mode = preempt_dynamint sched_dynamic_mode (const char *str) { -# ifndef CONFIG_PREEMPT_RT +# if !(defined(CONFIG_PREEMPT_RT ) || defined(CONFIG_ARCH_HAS_PREEMPT_LAZY ))if (!strcmp (str,"none" ))return preempt_dynamic_none;
kernel/sched/debug.c +1 -1
@@ -243,7 +243,7 @@ static ssize_t sched_dynamic_write(strucstatic int sched_dynamic_show (struct seq_file *m,void *v) { -int i =IS_ENABLED (CONFIG_PREEMPT_RT ) *2 ; +int i = (IS_ENABLED (CONFIG_PREEMPT_RT ) ||IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY )) *2 ;int j;/* Count entries in NULL terminated preempt_modes */
Quoted 24 lines · Peter Zijlstra
[ with 6.18 being an LTS release, it might be a good time for this ]
The introduction of
PREEMPT_LAZYwas for multiple reasons:-
PREEMPT_RTsuffered from over-scheduling, hurting performance compared to !PREEMPT_RT.- the introduction of (more) features that rely on preemption; like
folio_zero_user()which can do largememset()without preemption checks.(Xen already had a horrible hack to deal with long running hypercalls)
- the endless and uncontrolled sprinkling of
cond_resched()-- mostly cargo cult or in response to poor to replicate workloads.By moving to a model that is fundamentally preemptable these things become manageable and avoid needing to introduce more horrible hacks.
Since this is a requirement; limit
PREEMPT_NONEto architectures that do not support preemption at all. Further limitPREEMPT_VOLUNTARYto those architectures that do not yet havePREEMPT_LAZYsupport (with the eventual goal to make this the empty set and completely remove voluntary preemption andcond_resched()-- notably VOLUNTARY is already limited to !ARCH_NO_PREEMPT.)
Quoted 8 lines
This leaves up-to-date architectures (arm64, loongarch, powerpc, riscv, s390, x86) with only two preemption models: full and lazy (like
PREEMPT_RT).While Lazy has been the recommended setting for a while, not all distributions have managed to make the switch yet. Force things along. Keep the patch minimal in case of hard to address regressions that might pop up.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Steven Rostedt
maintainerkernel/Kconfig.preempt config ARCH_HAS_PREEMPT_LAZY · line 20
Quoted 8 lines · On Fri, 19 Dec 2025 11:15:02 +0100 Peter Zijlstra <[email protected]> wrote:
kernel/Kconfig.preempt +1
-0@@ -16,11 +16,13 @@ config ARCH_HAS_PREEMPT_LAZY choice prompt "Preemption Model" + default PREEMPT_LAZY if ARCH_HAS_PREEMPT_LAZY default PREEMPT_NONE
I think you can just make this:
default PREEMPT_LAZY
and remove the PREEMPT_NONE.
As PREEMPT_NONE now depends on ARCH_NO_PREEMPT and all the other options depend on !ARCH_NO_PREEMPT, the default will be PREEMPT_LAZY if it's available, but it will never be PREEMPT_NONE if it isn't unless PREEMPT_NONE is the only option available.
I added default PREEMPT_LAZY and did a:
$ mkdir /tmp/build $ make O=/tmp/build ARCH=alpha defconfig
And the result is:
CONFIG_PREEMPT_NONE_BUILD=yCONFIG_PREEMPT_NONE=y
-- Steve
Quoted 16 lines
kernel/Kconfig.preempt +2
-0config PREEMPT_NONE bool "No Forced Preemption (Server)" depends on !PREEMPT_RT + depends on ARCH_NO_PREEMPT select PREEMPT_NONE_BUILD if !PREEMPT_DYNAMIC help This is the traditional Linux preemption model, geared towards@@ -35,6 +37,7 @@ config PREEMPT_NONE config PREEMPT_VOLUNTARY bool "Voluntary Kernel Preemption (Desktop)" + depends on !ARCH_HAS_PREEMPT_LAZY depends on !ARCH_NO_PREEMPT depends on !PREEMPT_RT select PREEMPT_VOLUNTARY_BUILD if !PREEMPT_DYNAMIC
Hi Peter.
Quoted 77 lines · Peter Zijlstra
[ with 6.18 being an LTS release, it might be a good time for this ]
The introduction of
PREEMPT_LAZYwas for multiple reasons:-
PREEMPT_RTsuffered from over-scheduling, hurting performance compared to !PREEMPT_RT.- the introduction of (more) features that rely on preemption; like
folio_zero_user()which can do largememset()without preemption checks.(Xen already had a horrible hack to deal with long running hypercalls)
- the endless and uncontrolled sprinkling of
cond_resched()-- mostly cargo cult or in response to poor to replicate workloads.By moving to a model that is fundamentally preemptable these things become manageable and avoid needing to introduce more horrible hacks.
Since this is a requirement; limit
PREEMPT_NONEto architectures that do not support preemption at all. Further limitPREEMPT_VOLUNTARYto those architectures that do not yet havePREEMPT_LAZYsupport (with the eventual goal to make this the empty set and completely remove voluntary preemption andcond_resched()-- notably VOLUNTARY is already limited to !ARCH_NO_PREEMPT.)This leaves up-to-date architectures (arm64, loongarch, powerpc, riscv, s390, x86) with only two preemption models: full and lazy (like
PREEMPT_RT).While Lazy has been the recommended setting for a while, not all distributions have managed to make the switch yet. Force things along. Keep the patch minimal in case of hard to address regressions that might pop up.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/Kconfig.preempt | 3 +++ kernel/sched/core.c | 2 +- kernel/sched/debug.c | 2 +- 3 files changed, 5 insertions(+), 2 deletions(-)kernel/Kconfig.preempt +3
-0@@ -16,11 +16,13 @@ config ARCH_HAS_PREEMPT_LAZY choice prompt "Preemption Model" + default PREEMPT_LAZY if ARCH_HAS_PREEMPT_LAZY default PREEMPT_NONE config PREEMPT_NONE bool "No Forced Preemption (Server)" depends on !PREEMPT_RT + depends on ARCH_NO_PREEMPT select PREEMPT_NONE_BUILD if !PREEMPT_DYNAMIC help This is the traditional Linux preemption model, geared towards@@ -35,6 +37,7 @@ config PREEMPT_NONE config PREEMPT_VOLUNTARY bool "Voluntary Kernel Preemption (Desktop)" + depends on !ARCH_HAS_PREEMPT_LAZY depends on !ARCH_NO_PREEMPT depends on !PREEMPT_RT select PREEMPT_VOLUNTARY_BUILD if !PREEMPT_DYNAMICkernel/sched/core.c +1
-1@@ -7553,7 +7553,7 @@ int preempt_dynamic_mode = preempt_dynamint sched_dynamic_mode (const char *str) { -# ifndef CONFIG_PREEMPT_RT +# if !(defined(CONFIG_PREEMPT_RT ) || defined(CONFIG_ARCH_HAS_PREEMPT_LAZY ))if (!strcmp (str,"none" ))return preempt_dynamic_none;
kernel/sched/debug.c static ssize_t sched_dynamic_write(struc · line 249
Quoted 8 lines
kernel/sched/debug.c +1
-1@@ -243,7 +243,7 @@ static ssize_t sched_dynamic_write(strucstatic int sched_dynamic_show (struct seq_file *m,void *v) { -int i =IS_ENABLED (CONFIG_PREEMPT_RT ) *2 ; +int i = (IS_ENABLED (CONFIG_PREEMPT_RT ) ||IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY )) *2 ;int j;/* Count entries in NULL terminated preempt_modes */
Maybe only change the default to LAZY, but keep other options possible via dynamic update?
- When the kernel changes to lazy being the default, the scheduling pattern can change and it may affect the workloads. having ability to dynamically change to none/voluntary could help one to figure out where it is regressing. we could document cases where regression is expected.
- with preempt=full/lazy we will likely never see softlockups. How are we going to find out longer kernel paths(some maybe design, some may be bugs) apart from observing workload regression?
Also, is softlockup code is of any use in preempt=full/lazy?
Peter Zijlstra
maintainerQuoted 13 lines · Shrikanth Hegde
kernel/sched/debug.c static ssize_t sched_dynamic_write(struc · line 247
Quoted 9 lines
kernel/sched/debug.c +1
-1@@ -243,7 +243,7 @@ static ssize_t sched_dynamic_write(strucstatic int sched_dynamic_show (struct seq_file *m,void *v) { -int i =IS_ENABLED (CONFIG_PREEMPT_RT ) *2 ; +int i = (IS_ENABLED (CONFIG_PREEMPT_RT ) ||IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY )) *2 ;int j;/* Count entries in NULL terminated preempt_modes */ Maybe only change the default to LAZY, but keep other options possible via dynamic update?
Quoted 4 lines
- When the kernel changes to lazy being the default, the scheduling pattern can change and it may affect the workloads. having ability to dynamically change to none/voluntary could help one to figure out where
it is regressing. we could document cases where regression is expected.
I suppose we could do this. I just worry people will end up with 'echo volatile > /debug/sched/preempt' in their startup script, rather than trying to actually debug their issues.
Anybody with enough knowledge to be useful, can edit this line on their own, rebuild the kernel and go forth.
Also, I've already heard people are interested in compile-time removing of cond_resched() infrastructure for ARCH_HAS_PREEMPT_LAZY, so this would be short lived indeed.
Quoted 3 lines
- with preempt=full/lazy we will likely never see softlockups. How are we going to find out longer kernel
paths(some maybe design, some may be bugs)apart from observing workload regression?
Given the utter cargo cult placement of cond_resched(); I don't think we've actually lost much here. You wouldn't have seen the softlockup thing anyway, because of cond_resched().
Anyway, you can always build on top of function graph tracing, create a flame graph of stuff and see just where all your runtime went. I'm sure there's tools that do this already. Perhaps if you're handy with the BPF stuff you can even create a 'watchdog' of sorts that will scream if any function takes longer than X us to run or whatever.
Oh, that reminds me, Steve, would it make sense to have task_struct::se.sum_exec_runtime as a trace-clock?
Quoted 1 line
Also, is softlockup code is of any use in preempt=full/lazy?
Softlockup has always seemed of dubious value to me -- then again, I've been running preempt=y kernels from about the day that became an option :-)
I think it still trips if you loose a wakeup or something.
Quoted 15 lines · Peter Zijlstra
Quoted 13 lines · Shrikanth Hegde
kernel/sched/debug.c static ssize_t sched_dynamic_write(struc · line 247
Quoted 9 lines
kernel/sched/debug.c +1
-1@@ -243,7 +243,7 @@ static ssize_t sched_dynamic_write(strucstatic int sched_dynamic_show (struct seq_file *m,void *v) { -int i =IS_ENABLED (CONFIG_PREEMPT_RT ) *2 ; +int i = (IS_ENABLED (CONFIG_PREEMPT_RT ) ||IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY )) *2 ;int j;/* Count entries in NULL terminated preempt_modes */ Maybe only change the default to LAZY, but keep other options possible via dynamic update?
Quoted 8 lines
Quoted 4 lines
- When the kernel changes to lazy being the default, the scheduling pattern can change and it may affect the workloads. having ability to dynamically change to none/voluntary could help one to figure out where
it is regressing. we could document cases where regression is expected.I suppose we could do this. I just worry people will end up with 'echo volatile > /debug/sched/preempt' in their startup script, rather than trying to actually debug their issues.
Ack.
Quoted 25 lines
Anybody with enough knowledge to be useful, can edit this line on their own, rebuild the kernel and go forth.
Also, I've already heard people are interested in compile-time removing of
cond_resched()infrastructure forARCH_HAS_PREEMPT_LAZY, so this would be short lived indeed.Quoted 3 lines
- with preempt=full/lazy we will likely never see softlockups. How are we going to find out longer kernel
paths(some maybe design, some may be bugs)apart from observing workload regression?Given the utter cargo cult placement of
cond_resched(); I don't think we've actually lost much here. You wouldn't have seen the softlockup thing anyway, because ofcond_resched().Anyway, you can always build on top of function graph tracing, create a flame graph of stuff and see just where all your runtime went. I'm sure there's tools that do this already. Perhaps if you're handy with the BPF stuff you can even create a 'watchdog' of sorts that will scream if any function takes longer than X us to run or whatever.
Oh, that reminds me, Steve, would it make sense to have
task_struct::se.sum_exec_runtimeas a trace-clock?
Quoted 8 lines
Quoted 1 line
Also, is softlockup code is of any use in preempt=full/lazy?
Softlockup has always seemed of dubious value to me -- then again, I've been running preempt=y kernels from about the day that became an option :-)
I think it still trips if you loose a wakeup or something.
That's probably hungtask report right?
IIUC that would be independent of preemption model.
Steven Rostedt
maintainerQuoted 2 lines · On Wed, 25 Feb 2026 11:53:45 +0100 Peter Zijlstra <[email protected]> wrote:
Oh, that reminds me, Steve, would it make sense to have
task_struct::se.sum_exec_runtimeas a trace-clock?
That's unique per task right? As tracing is global it requires the clock to be monotonic, and I'm guessing a single sched_switch will break that.
Now if one wants to trace how long kernel paths are, I'm sure we could trivially make a new tracer to do so.
echo max_kernel_time > current_tracer
or something like that, that could act like a latency tracer that monitors how long any kernel thread runs without being preempted.
-- Steve
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · On Wed, 25 Feb 2026 11:53:45 +0100 Peter Zijlstra <[email protected]> wrote:
Oh, that reminds me, Steve, would it make sense to have
task_struct::se.sum_exec_runtimeas a trace-clock?
Quoted 8 lines
That's unique per task right? As tracing is global it requires the clock to be monotonic, and I'm guessing a single
sched_switchwill break that.Now if one wants to trace how long kernel paths are, I'm sure we could trivially make a new tracer to do so.
echo
max_kernel_time>current_tracer
That is good idea.
Quoted 5 lines
or something like that, that could act like a latency tracer that monitors how long any kernel thread runs without being preempted.
-- Steve
With preempt=full/lazy a long running kernel task can get preempted if it is running in preemptible section. that's okay.
My intent was to have a tracer that can say, look this kernel task took this much time before it completed. For some task such as long page walk, we know it is okay since it is expected to take time, but for some task such as reading watchdog shouldn't take time. But on large system's doing these global variable update itself may take a long time. Updating less often was a fix which had fixed that lockup IIRC. So how can we identify such opportunities. Hopefully I am making sense.
Earlier, one would have got a softlockup when things were making very slow progress(one's which didn't have a cond_resched)
Now, we don't know unless we see a workload regression.
If we don't have a tracer/mechanism today which gives kernel_tasks > timelimit, then having a new one would help.
Steven Rostedt
maintainerQuoted 11 lines · On Thu, 26 Feb 2026 11:00:14 +0530 Shrikanth Hegde <[email protected]>…
Quoted 10 lines · Steven Rostedt
Quoted 2 lines · On Wed, 25 Feb 2026 11:53:45 +0100 Peter Zijlstra <[email protected]> wrote:
Oh, that reminds me, Steve, would it make sense to have
task_struct::se.sum_exec_runtimeas a trace-clock?That's unique per task right? As tracing is global it requires the clock to be monotonic, and I'm guessing a single
sched_switchwill break that.
Quoted 6 lines
Quoted 4 lines
Now if one wants to trace how long kernel paths are, I'm sure we could trivially make a new tracer to do so.
echo
max_kernel_time>current_tracerThat is good idea.
Yeah, I think something like this should be added now that LAZY will prevent us from knowing where in the kernel is really going on for a long time.
Quoted 5 lines
Quoted 5 lines
or something like that, that could act like a latency tracer that monitors how long any kernel thread runs without being preempted.
Quoted 7 lines
Quoted 1 line
-- Steve
With preempt=full/lazy a long running kernel task can get preempted if it is running in preemptible section. that's okay.
My intent was to have a tracer that can say, look this kernel task took this much time before it completed. For some task such as long page walk, we know it is okay since
Tracers can be set to only watch a single task. The function and function graph tracers use set_ftrace_pid. I could extend that to other tracers. Hmm, that may even be useful for the preemptirq tracer!
Quoted 4 lines
it is expected to take time, but for some task such as reading watchdog shouldn't take time. But on large system's doing these global variable update itself may take a long time. Updating less often was a fix which had fixed that lockup IIRC. So how can we identify such opportunities. Hopefully I am making sense.
Not really. Can you explain in more detail, or specific examples of what constitutes a path you want to trace and one that you do not?
-- Steve
Hi Steven.
Quoted 17 lines · Steven Rostedt
Quoted 14 lines · On Thu, 26 Feb 2026 11:00:14 +0530 Shrikanth Hegde <[email protected]>…
Quoted 13 lines · Steven Rostedt
Quoted 2 lines · On Wed, 25 Feb 2026 11:53:45 +0100 Peter Zijlstra <[email protected]> wrote:
Oh, that reminds me, Steve, would it make sense to have
task_struct::se.sum_exec_runtimeas a trace-clock?That's unique per task right? As tracing is global it requires the clock to be monotonic, and I'm guessing a single
sched_switchwill break that.Now if one wants to trace how long kernel paths are, I'm sure we could trivially make a new tracer to do so.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
echo
max_kernel_time>current_tracerThat is good idea.
Yeah, I think something like this should be added now that LAZY will prevent us from knowing where in the kernel is really going on for a long time.
That would be the goal.
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
or something like that, that could act like a latency tracer that monitors how long any kernel thread runs without being preempted.
-- Steve
With preempt=full/lazy a long running kernel task can get preempted if it is running in preemptible section. that's okay.
Quoted 10 lines
Quoted 2 lines
My intent was to have a tracer that can say, look this kernel task took this much time before it completed. For some task such as long page walk, we know it is okay since
Tracers can be set to only watch a single task. The function and function graph tracers use
set_ftrace_pid. I could extend that to other tracers. Hmm, that may even be useful for the preemptirq tracer!Quoted 3 lines
it is expected to take time, but for some task such as reading watchdog shouldn't take time. But on large system's doing these global variable update itself may take a long time. Updating less often was a fix which had fixed that lockup IIRC. So how can we identify such
That was a hardlockup. wrong example.
Quoted 4 lines
Quoted 1 line
opportunities. Hopefully I am making sense.
Not really. Can you explain in more detail, or specific examples of what constitutes a path you want to trace and one that you do not?
All I was saying, there have been fixes which solved softlockup issues without using cond_resched. But seeing softlockup was important to know that issue existed.
Some reference commit I think that did this;a8c861f401b4 xfs: avoid busy loops in GCDe1b849cfa6b6 writeback: Avoid contention on wb->list_lock when switching inodes 0ddfb62f5d01 fix the softlockups in attach_recursive_mnt()
I am afraid we will have trace all functions to begin with (which is expensive), but filter out those which took minimal time (like less than a 1s or so). that would eventually leave only a few functions that actually took more than 1s(that should have limited overhead).
Steven Rostedt
maintainerQuoted 3 lines · On Fri, 27 Feb 2026 14:39:42 +0530 Shrikanth Hegde <[email protected]>…
I am afraid we will have trace all functions to begin with (which is expensive), but filter out those which took minimal time (like less than a 1s or so). that would eventually leave only a few functions that actually took more than 1s(that should have limited overhead).
Well, I think the detection can be done with timings between schedules. What's the longest running task without any voluntary schedule. Then you can add function graph tracing to it where it can possibly trigger in the location that detected the issue.
On a detection of a long schedule, a stack trace can be recorded. Using that stack trace, you could use the function graph tracer to see what is happening.
Anyway, something to think about, and this could be a topic at this years Linux Plumbers Tracing MC ;-)
-- Steve
Hi Steve.
Quoted 12 lines · Steven Rostedt
Quoted 3 lines · On Fri, 27 Feb 2026 14:39:42 +0530 Shrikanth Hegde <[email protected]>…
I am afraid we will have trace all functions to begin with (which is expensive), but filter out those which took minimal time (like less than a 1s or so). that would eventually leave only a few functions that actually took more than 1s(that should have limited overhead).
Well, I think the detection can be done with timings between schedules. What's the longest running task without any voluntary schedule. Then you can add function graph tracing to it where it can possibly trigger in the location that detected the issue.
Quoted 7 lines
On a detection of a long schedule, a stack trace can be recorded. Using that stack trace, you could use the function graph tracer to see what is happening.
Anyway, something to think about, and this could be a topic at this years Linux Plumbers Tracing MC ;-)
Yep. Will try to do this.
Someone from our tracing team wanted to give this a try too. Lets see.
Quoted 1 line
-- Steve
Quoted 10 lines · Shrikanth Hegde
Hi Steve.
Quoted 7 lines · Steven Rostedt
Quoted 3 lines · On Fri, 27 Feb 2026 14:39:42 +0530 Shrikanth Hegde <[email protected]>…
I am afraid we will have trace all functions to begin with (which is expensive), but filter out those which took minimal time (like less than a 1s or so). that would eventually leave only a few functions that actually took more than 1s(that should have limited overhead).
It is possible to remove tracing for some function after they were enabled in kernel? or this could only be done from user by looking at trace buffer?
Even if it doable, This would allow us to trace functions that took a lot of time. But we should be aiming to calculate the kernel paths that took a lot of time?
Quoted 5 lines
Quoted 5 lines
Well, I think the detection can be done with timings between schedules. What's the longest running task without any voluntary schedule. Then you can add function graph tracing to it where it can possibly trigger in the location that detected the issue.
This would not work either. We will have sched in/sched out even when running in userspace.
Lets say, user makes a syscall, the process will continue to in R state, we only need to track the long running time in kernel, but not in userspace.
Quoted 8 lines
Quoted 7 lines
On a detection of a long schedule, a stack trace can be recorded. Using that stack trace, you could use the function graph tracer to see what is happening.
Anyway, something to think about, and this could be a topic at this years Linux Plumbers Tracing MC ;-)
We could track the kernel paths, i.e different entry/exit points into kernel.
1. syscall entry/exit.
2. irq entry/exit.
3. kworker threads.
For 1 and 2 we have tracepoints already. For 3, we can use sched in/sched out tracepoints to see if and when it takes a long time.
All of them could be combined in one bpf program. Any thoughts?
Getting stacktrace of 3 is doable i guess, i.e when sched_out happen while in R state and time check has failed. But for 1,2 getting a stack is going to be difficult.
Please add if i have missed more kernel paths where we want detection to happen.
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 7dadeaa6e851e7d67733f3e24fc53ee107781d0f Gitweb: https://git.kernel.org/tip/7dadeaa6e851e7d67733f3e24fc53ee107781d0f Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 18 Dec 2025 15:25:10 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 08 Jan 2026 12:43:57 +01:00
sched: Further restrict the preemption modes
The introduction of PREEMPT_LAZY was for multiple reasons:
- PREEMPT_RT suffered from over-scheduling, hurting performance compared to !PREEMPT_RT.
- the introduction of (more) features that rely on preemption; like folio_zero_user() which can do large memset() without preemption checks.
(Xen already had a horrible hack to deal with long running hypercalls)
- the endless and uncontrolled sprinkling of cond_resched() -- mostly cargo cult or in response to poor to replicate workloads.
By moving to a model that is fundamentally preemptable these things become managable and avoid needing to introduce more horrible hacks.
Since this is a requirement; limit PREEMPT_NONE to architectures that do not support preemption at all. Further limit PREEMPT_VOLUNTARY to those architectures that do not yet have PREEMPT_LAZY support (with the eventual goal to make this the empty set and completely remove voluntary preemption and cond_resched() -- notably VOLUNTARY is already limited to !ARCH_NO_PREEMPT.)
This leaves up-to-date architectures (arm64, loongarch, powerpc, riscv, s390, x86) with only two preemption models: full and lazy.
While Lazy has been the recommended setting for a while, not all distributions have managed to make the switch yet. Force things along. Keep the patch minimal in case of hard to address regressions that might pop up.
--- kernel/Kconfig.preempt | 3 +++ kernel/sched/core.c | 2 +- kernel/sched/debug.c | 2 +- 3 files changed, 5 insertions(+), 2 deletions(-)
kernel/Kconfig.preempt +3 -0
@@ -16,11 +16,13 @@ config ARCH_HAS_PREEMPT_LAZY choice prompt "Preemption Model" + default PREEMPT_LAZY if ARCH_HAS_PREEMPT_LAZY default PREEMPT_NONE config PREEMPT_NONE bool "No Forced Preemption (Server)" depends on !PREEMPT_RT + depends on ARCH_NO_PREEMPT select PREEMPT_NONE_BUILD if !PREEMPT_DYNAMIC help This is the traditional Linux preemption model, geared towards
@@ -35,6 +37,7 @@ config PREEMPT_NONE config PREEMPT_VOLUNTARY bool "Voluntary Kernel Preemption (Desktop)" + depends on !ARCH_HAS_PREEMPT_LAZY depends on !ARCH_NO_PREEMPT depends on !PREEMPT_RT select PREEMPT_VOLUNTARY_BUILD if !PREEMPT_DYNAMIC
kernel/sched/core.c +1 -1
@@ -7553,7 +7553,7 @@ int preempt_dynamic_mode = preempt_dynamic_undefined;int sched_dynamic_mode (const char *str) { -# ifndef CONFIG_PREEMPT_RT +# if !(defined(CONFIG_PREEMPT_RT ) || defined(CONFIG_ARCH_HAS_PREEMPT_LAZY ))if (!strcmp (str,"none" ))return preempt_dynamic_none;
kernel/sched/debug.c +1 -1
@@ -243,7 +243,7 @@ static ssize_t sched_dynamic_write(struct file *filp, const char __user *ubuf,static int sched_dynamic_show (struct seq_file *m,void *v) { -int i =IS_ENABLED (CONFIG_PREEMPT_RT ) *2 ; +int i = (IS_ENABLED (CONFIG_PREEMPT_RT ) ||IS_ENABLED (CONFIG_ARCH_HAS_PREEMPT_LAZY )) *2 ;int j;/* Count entries in NULL terminated preempt_modes */
On 19/12/2025 10:15, Peter Zijlstra wrote:
Hi Peter,
We are observing a performance regression on s390 since enabling PREEMPT_LAZY. Test Environment
Architecture: s390
Setup:
Single KVM host running two identical guests
Guests are connected virtually via Open vSwitch
Workload: uperf streaming read test with 50 parallel connections One guest acts as the uperf client, the other as the server
Open vSwitch configuration:
OVS bridge with two ports
Guests attached via virtio‑net
Each guest configured with 4 vhost‑queues
Problem Description
When comparing PREEMPT_LAZY against full PREEMPT, we see a substantial drop in throughput—on some systems up to 50%.
Observed Behaviour
By tracing packets inside Open vSwitch (ovs_do_execute_action), we see:
Packet drops
Retransmissions
Reductions in packet size (from 64K down to 32K)
Capturing traffic inside the VM and inspecting it in Wireshark shows the following TCP‑level differences between PREEMPT_FULL and PREEMPT_LAZY:
|--------------------------------------+--------------+--------------+------------------| | Wireshark Warning / Note | PREEMPT_FULL | PREEMPT_LAZY | (lazy vs full) | |--------------------------------------+--------------+--------------+------------------| | D-SACK Sequence | 309 | 2603 | ×8.4 | | Partial Acknowledgement of a segment | 54 | 279 | ×5.2 | | Ambiguous ACK (Karn) | 32 | 747 | ×23 | | (Suspected) spurious retransmission | 205 | 857 | ×4.2 | | (Suspected) fast retransmission | 54 | 1622 | ×30 | | Duplicate ACK | 504 | 3446 | ×6.8 | | Packet length exceeds MSS (TSO/GRO) | 13172 | 34790 | ×2.6 | | Previous segment(s) not captured | 9205 | 6730 | -27% | | ACKed segment that wasn't captured | 7022 | 8272 | +18% | | (Suspected) out-of-order segment | 436 | 303 | -31% | |--------------------------------------+--------------+--------------+------------------|
This pattern indicates reordering, loss, or scheduling‑related delays, but it is still unclear why PREEMPT_LAZY is causing this behaviour in this workload.
Additional observations:
Monitoring the guest CPU run time shows that it drops from 16% with PREEMPT_FULL to 9% with PREEMPT_LAZY.
The workload is dominated by voluntary preemption (schedule()), and PREEMPT_LAZY is, as far as I understand, mainly concerned with forced preemption. It is therefore not obvious why PREEMPT_LAZY has an impact here.
Changing guest configuration to disable mergeable RX buffers:
<host mrg_rxbuf="off"/>
had a clear effect on throughput:
PREEMPT_LAZY: throughput improved from 40 Gb/s → 60 Gb/s
Sebastian Andrzej Siewior
maintainerQuoted 7 lines · Ciunas Bennett
Monitoring the guest CPU run time shows that it drops from 16% with
PREEMPT_FULLto 9% withPREEMPT_LAZY.The workload is dominated by voluntary preemption (
schedule()), andPREEMPT_LAZYis, as far as I understand, mainly concerned with forced preemption.
It is therefore not obvious whyPREEMPT_LAZYhas an impact here.
PREEMPT_FULL schedules immediately if there is a preemption request either due to a wake up of a task, or because the time slice is used up (while in kernel).PREEMPT_LAZY delays the preemption request, caused by the scheduling event, either until the task returns to userland or the next HZ tick.
The voluntary schedule() invocation shouldn't be effected by FULL-> LAZY but I guess FULL scheduled more often after a wake up which is in favour.
Quoted 5 lines
Changing guest configuration to disable mergeable RX buffers:
<host mrg_rxbuf="off"/>
had a clear effect on throughput:
PREEMPT_LAZY: throughput improved from 40 Gb/s → 60 Gb/s
Brings this the workload/ test to PREEMPT_FULL level?
Sebastian
Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 5 lines · Ciunas Bennett
Changing guest configuration to disable mergeable RX buffers:
<host mrg_rxbuf="off"/>
had a clear effect on throughput:
PREEMPT_LAZY: throughput improved from 40 Gb/s → 60 Gb/sBrings this the workload/ test to
PREEMPT_FULLlevel?
Sorry was not clear here, so when I enable this there is also an improvement in PREEMPT_FULL from 55Gb/s -> 60Gb/s
So I see an improvement in both test cases.PREEMPT_LAZY: throughput improved from 40 Gb/s → 60 Gb/sPREEMPT_FULL: throughput improved from 55 Gb/s → 60 Gb/s
Quoted 58 lines · Ciunas Bennett
On 19/12/2025 10:15, Peter Zijlstra wrote:
Hi Peter,
We are observing a performance regression on s390 since enablingPREEMPT_LAZY. Test Environment
Architecture: s390
Setup:Single KVM host running two identical guests
Guests are connected virtually via Open vSwitch
Workload: uperf streaming read test with 50 parallel connections One guest acts as the uperf client, the other as the serverOpen vSwitch configuration:
OVS bridge with two ports
Guests attached via virtio‑net
Each guest configured with 4 vhost‑queuesProblem Description
When comparingPREEMPT_LAZYagainst full PREEMPT, we see a substantial drop in throughput—on some systems up to 50%.Observed Behaviour
By tracing packets inside Open vSwitch (ovs_do_execute_action), we see:
Packet drops
Retransmissions
Reductions in packet size (from 64K down to 32K)Capturing traffic inside the VM and inspecting it in Wireshark shows the following TCP‑level differences between
PREEMPT_FULLandPREEMPT_LAZY:
|--------------------------------------+--------------+--------------+------------------|| Wireshark Warning / Note | PREEMPT_FULL | PREEMPT_LAZY | (lazy vs full) | |--------------------------------------+--------------+--------------+------------------|| D-SACK Sequence | 309 | 2603 | ×8.4 | | Partial Acknowledgement of a segment | 54 | 279 | ×5.2 | | Ambiguous ACK (Karn) | 32 | 747 | ×23 | | (Suspected) spurious retransmission | 205 | 857 | ×4.2 | | (Suspected) fast retransmission | 54 | 1622 | ×30 | | Duplicate ACK | 504 | 3446 | ×6.8 | | Packet length exceeds MSS (TSO/GRO) | 13172 | 34790 | ×2.6 | | Previous segment(s) not captured | 9205 | 6730 | -27% | | ACKed segment that wasn't captured | 7022 | 8272 | +18% | | (Suspected) out-of-order segment | 436 | 303 | -31% | |--------------------------------------+--------------+--------------+------------------|This pattern indicates reordering, loss, or scheduling‑related delays, but it is still unclear why
PREEMPT_LAZYis causing this behaviour in this workload.Additional observations:
Monitoring the guest CPU run time shows that it drops from 16% with
PREEMPT_FULLto 9% withPREEMPT_LAZY.
Quoted 7 lines
The workload is dominated by voluntary preemption (
schedule()), andPREEMPT_LAZYis, as far as I understand, mainly concerned with forced preemption. It is therefore not obvious whyPREEMPT_LAZYhas an impact here.Changing guest configuration to disable mergeable RX buffers:
<host mrg_rxbuf="off"/>
had a clear effect on throughput:
PREEMPT_LAZY: throughput improved from 40 Gb/s → 60 Gb/s
When I look at top sched_switch kstacks on s390 with this workload, 20% of them are worker_thread() -> schedule(), both with CONFIG_PREEMPT and CONFIG_PREEMPT_LAZY. The others are vhost and idle.
On x86 I see only vhost and idle, but not worker_thread().
According to runqlat.bt, average run queue latency goes up from 4us to 18us when switching from CONFIG_PREEMPT to CONFIG_PREEMPT_LAZY.
I modified the script to show per-comm latencies, and it shows that worker_thread() is disproportionately penalized: the latency increases from 2us to 60us!
For vhost it's better: 5us -> 2us, and for KVM it's better too: 8us -> 2us.
Finally, what is the worker doing? I looked at __queue_work() kstacks, and they all come from irqfd_wakeup().
irqfd_wakeup() calls arch-specific kvm_arch_set_irq_inatomic(), which is implemented on x86 and not implemented on s390.
This may explain why we on s390 are the first to see this.
Christian, do you think if it would make sense to implement kvm_arch_set_irq_inatomic() on s390?
Christian Borntraeger
reviewerAm 24.02.26 um 21:30 schrieb Ilya Leoshkevich:
Quoted 9 lines
Finally, what is the worker doing? I looked at
__queue_work()kstacks, and they all come fromirqfd_wakeup().
irqfd_wakeup()calls arch-specifickvm_arch_set_irq_inatomic(), which is implemented on x86 and not implemented on s390.This may explain why we on s390 are the first to see this.
Christian, do you think if it would make sense to implement kvm_arch_set_irq_inatomic() on s390?
So in fact Doug is working on that at the moment. There are some corner cases where we had concerns as we have to pin the guest pages holding the interrupt bits. This was secure execution, I need to followup if we have already solved those cases. But we can try if the current patch will help this particular problem.
If yes, then we can try to speed up the work on this.
Christian
Quoted 17 lines · Christian Borntraeger
Am 24.02.26 um 21:30 schrieb Ilya Leoshkevich:
Quoted 9 lines
Finally, what is the worker doing? I looked at
__queue_work()kstacks, and they all come fromirqfd_wakeup().
irqfd_wakeup()calls arch-specifickvm_arch_set_irq_inatomic(), which is implemented on x86 and not implemented on s390.This may explain why we on s390 are the first to see this.
Christian, do you think if it would make sense to implement kvm_arch_set_irq_inatomic() on s390?
So in fact Doug is working on that at the moment. There are some corner cases where we had concerns as we have to pin the guest pages holding the interrupt bits. This was secure execution, I need to followup if we have already solved those cases. But we can try if the current patch will help this particular problem.
Quoted 3 lines
If yes, then we can try to speed up the work on this.
Christian
Christian, the patch is very close to ready. The last step, I rebased on Master today to pickup the latest changes to interrupt.c. I am building that now and will test for non-SE and SE environments. I have been testing my solution for SE environments for a few weeks and it seems to cover the use cases I have tested.
A quick update on the issue.
Introducing kvm_arch_set_irq_inatomic() appears to make the problem go away on my setup.
That said, this still begs the question: why does irqfd_wakeup behave differently (or poorly) in this scenario compared to the in-atomic IRQ injection path? Is there a known interaction with workqueues, contexts, or locking that would explain the divergence here?
Observations:irqfd_wakeup: triggers the problematic behaviour.
Forcing in-atomic IRQ injection (kvm_arch_set_irq_inatomic): issue not observed.
@Peter Zijlstra — Peter, do you have thoughts on how the workqueue scheduling context here could differ enough to cause this regression? Any pointers on what to trace specifically in irqfd_wakeup and the work item path would be appreciated.
Thanks,
Ciunas Bennett
Quoted 4 lines · Douglas Freimuth
Christian, the patch is very close to ready. The last step, I rebased on Master today to pickup the latest changes to
interrupt.c. I am building that now and will test for non-SE and SE environments. I have been testing my solution for SE environments for a few weeks and it seems to cover the use cases I have tested.
Peter Zijlstra
maintainerQuoted 11 lines · Ciunas Bennett
A quick update on the issue.
Introducingkvm_arch_set_irq_inatomic()appears to make the problem go away on my setup.
That said, this still begs the question: why doesirqfd_wakeupbehave differently (or poorly) in this scenario compared to the in-atomic IRQ injection path? Is there a known interaction with workqueues, contexts, or locking that would explain the divergence here?Observations:
irqfd_wakeup: triggers the problematic behaviour.
Forcing in-atomic IRQ injection (kvm_arch_set_irq_inatomic): issue not observed.@Peter Zijlstra — Peter, do you have thoughts on how the workqueue scheduling context here could differ enough to cause this regression? Any pointers on what to trace specifically in
irqfd_wakeupand the work item path would be appreciated.
So the thing that LAZY does different from FULL is that it delays preemption a bit.
This has two ramifications:
1) some ping-pong workloads will turn into block+wakeup, adding overhead.
FULL: running your task A, an interrupt would come in, wake task B and set Need Resched and the interrupt return path calls schedule() and you're task B. B does its thing, 'wakes' A and blocks.
LAZY: running your task A, an interrupt would come in, wake task B (no NR set), you continue running A, A blocks for it needs something of B, now you schedule() [*] B runs, does its thing, does an actual wakeup of A and blocks.
The distinct difference here is that LAZY does a block of A and consequently B has to do a full wakeup of A, whereas FULL doesn't do a block of A, and hence the wakeup of A is NOP as well.
2) Since the schedule() is delayed, it might happen that by the time it does get around to it, your task B is no longer the most eligible option.
Same as above, except now, C is also woken, and the schedule marked with [*] picks C, this then results in a detour, delaying things further.
Quoted 19 lines · Peter Zijlstra
This has two ramifications:
1) some ping-pong workloads will turn into block+wakeup, adding overhead.
FULL: running your task A, an interrupt would come in, wake task B and set Need Resched and the interrupt return path calls
schedule()and you're task B. B does its thing, 'wakes' A and blocks.LAZY: running your task A, an interrupt would come in, wake task B (no NR set), you continue running A, A blocks for it needs something of B, now you
schedule()[*] B runs, does its thing, does an actual wakeup of A and blocks.The distinct difference here is that LAZY does a block of A and consequently B has to do a full wakeup of A, whereas FULL doesn't do a block of A, and hence the wakeup of A is NOP as well.
Quoted 9 lines
2) Since the
schedule()is delayed, it might happen that by the time it does get around to it, your task B is no longer the most eligible option.Same as above, except now, C is also woken, and the schedule marked with [*] picks C, this then results in a detour, delaying things further.
Hi Peter,
I wanted to share an update/findings from the investigations that I carried out for the issue mentioned above.
Quick refresh:
Workload: uperf sending TCP data between two VMs (client and server), each configured with a single vhost queue (min vhost ques for testing)
Issue: With lazy preemption as the default preemption mode where previously it was full preemption, there is a significant drop in performance for this workload
Simplification of the issue
We have two tasks:
TaskA produces data
TaskB consumes the data produced by TaskA
Notification path: TaskA informs TaskB that new data is available by adding a new item to a workqueue. This triggers a kworker which runs and notifies TaskB.
Issue
TaskA is configured to use schedule_work(). Internally, schedule_work() uses system_percpu_wq, which is configured as:
<WQ_PERCPU = 1 << 8, /* bound to a specific cpu */>
This means the workqueue item will be woken up and executed on the same CPU that queued the work.
If the task that queues the work (TaskA) is a long-running task with limited opportunities to call schedule(), then the kworker may be delayed significantly before it gets CPU time. In our scenario:
TaskA continuously produces data
There is no dependency requiring TaskA to yield due to TaskB
As a result, TaskA can occupy the CPU for an entire tick before being preempted by the kworker
Observed behavior
This is exactly what we observe in practice:
TaskB corresponds to the VM consuming data generated by our vhost task When running uperf, this behavior leads to a significant drop in throughput (Gb/s) The VM is unable to consume data in a timely manner
When it is finally notified of new data, the delayed signaling introduces jitter This causes TCP issues, including retransmissions and out-of-order packets
Results:
|--------------+-----+------------------+------------------------| | preempt mode | Gbs | workqueue pool | kworker latency avg ms | |--------------+-----+------------------+------------------------| | full | ~50 | system_percpu_wq | 0.002 | | lazy | ~13 | system_percpu_wq | 0.721 | | lazy | ~50 | system_dfl_wq | 0.005 | |--------------+-----+------------------+------------------------|
So I did some more testing and if I use a different workqueue pool the system_dfl_wq the TP was good again, as you can see in the results table. Since the kworker is not CPU-bound, the scheduler has flexibility to select a more suitable CPU for execution.
/* system_dfl_wq is unbound workqueue. Workers are not bound to
* any specific CPU, not concurrency managed, and all queued works are
* executed immediately as long as max_active limit is not reached and
* resources are available. */
Given this understanding, what would be the best approach here? Should we consider changing the workqueue usage in the KVM code, or do you see an alternative way to address this issue?
Sebastian Andrzej Siewior
maintainerOn 2026-06-05 11:43:24 [+0100], Ciunas Bennett wrote:
…
Quoted 14 lines
Quick refresh:
Workload: uperf sending TCP data between two VMs (client and server), each configured with a single vhost queue (min vhost ques for testing)
Issue: With lazy preemption as the default preemption mode where previously it was full preemption, there is a significant drop in performance for this workloadSimplification of the issue
We have two tasks:TaskA produces data
TaskB consumes the data produced by TaskANotification path: TaskA informs TaskB that new data is available by adding a new item to a workqueue. This triggers a kworker which runs and notifies TaskB.
Quoted 8 lines
Issue
TaskA is configured to useschedule_work(). Internally,schedule_work()usessystem_percpu_wq, which is configured as:
<WQ_PERCPU= 1 << 8, /* bound to a specific cpu */>This means the workqueue item will be woken up and executed on the same CPU that queued the work. If the task that queues the work (TaskA) is a long-running task with
limited opportunities to callschedule(), then the kworker may be
delayed significantly before it gets CPU time.
There is some work done by Marco to rework the API to explicitly state if a per-CPU workqueue is mandatory _or_ if an CPU unbound workqueue can be used instead. (Rather than having schedule_work() not knowing the implications).
Quoted 27 lines
In our scenario:
TaskA continuously produces data
There is no dependency requiring TaskA to yield due to TaskB
As a result, TaskA can occupy the CPU for an entire tick before being preempted by the kworkerObserved behavior
This is exactly what we observe in practice:TaskB corresponds to the VM consuming data generated by our vhost task When running uperf, this behavior leads to a significant drop in throughput (Gb/s) The VM is unable to consume data in a timely manner
When it is finally notified of new data, the delayed signaling introduces jitter This causes TCP issues, including retransmissions and out-of-order packetsResults:
|--------------+-----+------------------+------------------------| | preempt mode | Gbs | workqueue pool | kworker latency avg ms | |--------------+-----+------------------+------------------------| | full | ~50 | system_percpu_wq | 0.002 | | lazy | ~13 | system_percpu_wq | 0.721 | | lazy | ~50 | system_dfl_wq | 0.005 | |--------------+-----+------------------+------------------------|So I did some more testing and if I use a different workqueue pool the
system_dfl_wqthe TP was good again, as you can see in the results table. Since the kworker is not CPU-bound, the scheduler has flexibility to select a more suitable CPU for execution.
Quoted 8 lines
/*
system_dfl_wqis unbound workqueue. Workers are not bound to
* any specific CPU, not concurrency managed, and all queued works are
* executed immediately as long asmax_activelimit is not reached and
* resources are available. */Given this understanding, what would be the best approach here? Should we consider changing the workqueue usage in the KVM code, or do you see an alternative way to address this issue?
It seems that using an unbound worker would avoid the problem at hand, correct?
Sebastian
Quoted 7 lines
Quoted 2 lines
delayed significantly before it gets CPU time.
There is some work done by Marco to rework the API to explicitly state if a per-CPU workqueue is mandatory _or_ if an CPU unbound workqueue can be used instead. (Rather than having
schedule_work()not knowing the implications).
Oh, that is good some work already being done here.
I had a small patch ready to go, but possibly the solution needs to be more robust.
- schedule_work(&irqfd->inject);
+ queue_work(system_dfl_wq, &irqfd->inject);
Quoted 4 lines
It seems that using an unbound worker would avoid the problem at hand, correct?
Sebastian
Yeah, changing to an unbound kworker avoids the issue in any of the testing I have done. Ciunas