CPU Lockups in KVM with deferred hrtimer rearming
Subsystems (8)
Hi Peter,
We noticed a KVM Unit test 'x2apic' - (APIC LVT timer one shot) failing, and also some TDX specific tests running into multiple CPUs in hard lockups on a 192-CPU Emerald Rapids system, and we traced it to the htrimers deferred rearming merge.
Making CONFIG_HRTIMER_REARM_DEFERRED default to n in Kconfig made both pass.
This is the hard lockup splat:
watchdog: CPU98: Watchdog detected hard LOCKUP on cpu 98
Modules linked in: openvswitch nsh tls ipt_REJECT iptable_mangle iptable_nat iptable_filter ip_tables bridge stp llc kvm_intel kvm irqbypass sunrpc
irq event stamp: 34998
hardirqs last enabled at (34997): [<ffffffffc090ce6d>] tdx_vcpu_run+0x5d/0x350 [kvm_intel]
hardirqs last disabled at (34998): [<ffffffffb9add6df>] exc_nmi+0xaf/0x1a0
softirqs last enabled at (34404): [<ffffffffb83fdd93>] __irq_exit_rcu+0xe3/0x160
softirqs last disabled at (34395): [<ffffffffb83fdd93>] __irq_exit_rcu+0xe3/0x160
CPU: 98 UID: 0 PID: 54785 Comm: qemu-system-x86 Not tainted 7.0.0-g10324ed6a556 #1 PREEMPT(full)
Hardware name: HPE ProLiant DL380 Gen11/ProLiant DL380 Gen11, BIOS 2.48 03/11/2025
RIP: 0010:vmx_do_nmi_irqoff+0x13/0x20 [kvm_intel]
Code: ff ff 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 48 83 e4 f0 6a 18 55 9c 6a 10 e8 3d db 6e f7 <c9> c3 cc cc cc cc 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90
RSP: 0018:ff8d3a069bdf3af0 EFLAGS: 00000086
RAX: ff3cc96963d68000 RBX: ff3cc96963d68000 RCX: 4000000200000000
RDX: 0000000080000200 RSI: ff3cc96963d699d0 RDI: ff3cc96963d68000
RBP: ff8d3a069bdf3af0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ff3cc968d03d0000 R14: ff3cc968d03d0000 R15: 0000000000000000
FS: 00007f26ab7fe6c0(0000) GS:ff3cc98782d76000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000001544af004 CR4: 0000000000f73ef0
PKRU: 00000000
Call Trace:
<TASK>
vmx_handle_nmi+0xdf/0x140 [kvm_intel]
tdx_vcpu_enter_exit+0xd5/0x300 [kvm_intel]
tdx_vcpu_run+0x5d/0x350 [kvm_intel]
vcpu_run+0xd4a/0x1800 [kvm]
? __local_bh_enable_ip+0x7b/0xf0
? kvm_arch_vcpu_ioctl_run+0x38b/0x5f0 [kvm]
? kvm_arch_vcpu_ioctl_run+0xb9/0x5f0 [kvm]
kvm_arch_vcpu_ioctl_run+0x38b/0x5f0 [kvm]
kvm_vcpu_ioctl+0x2ef/0xb00 [kvm]
? __fget_files+0x2b/0x190
? find_held_lock+0x2b/0x80
__x64_sys_ioctl+0x97/0xe0
do_syscall_64+0xf4/0x1540
? __x64_sys_ioctl+0xb1/0xe0
? trace_hardirqs_on_prepare+0xd2/0xf0
? do_syscall_64+0x225/0x1540
? trace_hardirqs_on+0x18/0x100
? __local_bh_enable_ip+0x7b/0xf0
? arch_do_signal_or_restart+0x155/0x250
? trace_hardirqs_off+0x4e/0xf0
? exit_to_user_mode_loop+0x150/0x4e0
? trace_hardirqs_on_prepare+0xd2/0xf0
? do_syscall_64+0x225/0x1540
? do_user_addr_fault+0x36c/0x6b0
? lockdep_hardirqs_on_prepare+0xdb/0x190
? trace_hardirqs_on+0x18/0x100
? do_syscall_64+0xab/0x1540
? exc_page_fault+0x12c/0x2b0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f45f7ae00ed
Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
RSP: 002b:00007f26ab7f3e70 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f26ab7fe6c0 RCX: 00007f45f7ae00ed
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000099
RBP: 00007f26ab7f3ec0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f26ab7fe6c0
R13: 00007ffdc7adecd0 R14: 00007f26ab7fecdc R15: 00007ffdc7adedd7
</TASK>I tried out AI assisted and patch (below) which does happen to solve it, but I'm not familiar in this area, and not sure if this is the right fix.
---
include/linux/entry-virt.h +2 -0
@@ -4,6 +4,7 @@#include <linux/static_call_types.h> #include <linux/resume_user_mode.h> +#include <linux/hrtimer_rearm.h> #include <linux/syscalls.h> #include <linux/seccomp.h> #include <linux/sched.h>
@@ -58,6 +59,7 @@ int xfer_to_guest_mode_handle_work(void);static inline void xfer_to_guest_mode_prepare (void ) {lockdep_assert_irqs_disabled (); +hrtimer_rearm_deferred ();tick_nohz_user_enter_prepare (); }
kernel/time/hrtimer.c +1 -0
@@ -2058,6 +2058,7 @@ void __hrtimer_rearm_deferred(void) }hrtimer_rearm (cpu_base, expires_next, true); } +EXPORT_SYMBOL_GPL (__hrtimer_rearm_deferred); static __always_inline voidhrtimer_interrupt_rearm (struct hrtimer_cpu_base *cpu_base,ktime_t expires_next)
Thomas Gleixner
maintainerQuoted 17 lines · Vishal L. Verma
I tried out AI assisted and patch (below) which does happen to solve it, but I'm not familiar in this area, and not sure if this is the right fix.
include/linux/entry-virt.h +1
-0@@ -4,6 +4,7 @@#include <linux/static_call_types.h> #include <linux/resume_user_mode.h> +#include <linux/hrtimer_rearm.h> #include <linux/syscalls.h> #include <linux/seccomp.h> #include <linux/sched.h>
include/linux/entry-virt.h int xfer_to_guest_mode_handle_work(void); · line 63
Quoted 5 lines
include/linux/entry-virt.h +1
-0@@ -58,6 +59,7 @@ int xfer_to_guest_mode_handle_work(void);static inline void xfer_to_guest_mode_prepare (void ) {lockdep_assert_irqs_disabled (); +hrtimer_rearm_deferred ();tick_nohz_user_enter_prepare ();
This code should never be reached with a rearm pending. Something else went wrong earlier. So while the patch "works" it papers over the underlying problem.
Can you please do the following:
1) Apply the patch below
2) Enable function tracing and the hrtimer* trace events
3) Enable tracing if it has been disabled already
echo 1 >/sys/kernel/tracing/tracing_on
4) Run the tests and wait for /sys/kernel/tracing/tracing_on to become 0, which means the problem triggered.
5) Retrieve the trace from /sys/kernel/tracing/trace and provide it somewhere to download from or send it to me compressed offlist.
Thanks,
tglx
---
include/linux/entry-virt.h +4 -0
@@ -58,6 +58,10 @@ int xfer_to_guest_mode_handle_work(void);static inline void xfer_to_guest_mode_prepare (void ) {lockdep_assert_irqs_disabled (); +if (test_thread_flag (TIF_HRTIMER_REARM )) { +tracing_off (); +hrtimer_rearm_deferred (); + }tick_nohz_user_enter_prepare (); }
Thomas Gleixner
maintainerQuoted 4 lines · Thomas Gleixner
On Thu, Apr 16 2026 at 20:50, Vishal L. Verma wrote:
This code should never be reached with a rearm pending. Something else went wrong earlier. So while the patch "works" it papers over the underlying problem.
Peter just noticed that this should be fixed with
1f5ffc672165 ("Fix mismerge of the arm64 / timer-core interrupt handling changes")
Thanks,
tglx
Quoted 13 lines · Thomas Gleixner
This code should never be reached with a rearm pending. Something else went wrong earlier. So while the patch "works" it papers over the underlying problem.
Can you please do the following:
1) Apply the patch below
2) Enable function tracing and the hrtimer* trace events
3) Enable tracing if it has been disabled already
Quoted 7 lines
echo 1 >/sys/kernel/tracing/tracing_on
4) Run the tests and wait for
/sys/kernel/tracing/tracing_onto become 0, which means the problem triggered.5) Retrieve the trace from
/sys/kernel/tracing/traceand provide it somewhere to download from or send it to me compressed offlist.
Hi Thomas,
I've uploaded the trace here (~75MB compressed):
https://drive.proton.me/urls/B9PY61XQ0C#07XwTVhE46eB
As for:
1f5ffc672165 ("Fix mismerge of the arm64 / timer-core interrupt handling changes")
I already had that commit in the branch that was tested and it didn't fix it.
Thomas Gleixner
maintainerQuoted 27 lines · Vishal L
Quoted 13 lines · Thomas Gleixner
This code should never be reached with a rearm pending. Something else went wrong earlier. So while the patch "works" it papers over the underlying problem.
Can you please do the following:
1) Apply the patch below
2) Enable function tracing and the hrtimer* trace events
3) Enable tracing if it has been disabled already
Quoted 7 lines
echo 1 >/sys/kernel/tracing/tracing_on
4) Run the tests and wait for
/sys/kernel/tracing/tracing_onto become 0, which means the problem triggered.5) Retrieve the trace from
/sys/kernel/tracing/traceand provide it somewhere to download from or send it to me compressed offlist.Hi Thomas,
I've uploaded the trace here (~75MB compressed):
https://drive.proton.me/urls/B9PY61XQ0C#07XwTVhE46eB
Quoted 6 lines
As for:
1f5ffc672165("Fix mismerge of the arm64 / timer-core interrupt handling changes")I already had that commit in the branch that was tested and it didn't fix it.
Thanks for the update. Can you try to provide the information I asked for above?
Thanks,
tglx
Quoted 30 lines · Thomas Gleixner
Quoted 29 lines · Vishal L
Quoted 13 lines · Thomas Gleixner
This code should never be reached with a rearm pending. Something else went wrong earlier. So while the patch "works" it papers over the underlying problem.
Can you please do the following:
1) Apply the patch below
2) Enable function tracing and the hrtimer* trace events
3) Enable tracing if it has been disabled already
Quoted 7 lines
echo 1 >/sys/kernel/tracing/tracing_on
4) Run the tests and wait for
/sys/kernel/tracing/tracing_onto become 0, which means the problem triggered.5) Retrieve the trace from
/sys/kernel/tracing/traceand provide it somewhere to download from or send it to me compressed offlist.Hi Thomas,
I've uploaded the trace here (~75MB compressed):
https://drive.proton.me/urls/B9PY61XQ0C#07XwTVhE46eBAs for:
Quoted 8 lines
Quoted 4 lines
1f5ffc672165("Fix mismerge of the arm64 / timer-core interrupt handling changes")I already had that commit in the branch that was tested and it didn't fix it.
Thanks for the update. Can you try to provide the information I asked for above?
Ah sorry - I should've said that with your patch applied, tracing_on did become 0, so the problem was triggered.
The trace from that is in the URL above.
This is how I collected it:
tracefs=/sys/kernel/tracing
echo 4096 > "$tracefs"/buffer_size_kb
echo function > "$tracefs"/current_tracer
echo 1 > "$tracefs"/events/hrtimer/enable
echo 1 > "$tracefs"/tracing_on
<run the test>
tracing_on="$(cat "$tracefs"/tracing_on)"
if [ "$tracing_on" -eq 0 ]; then
echo "Debug patch triggered, collecting trace" cat "$tracefs"/trace | gzip > /tmp/hrtimer_rearm_trace.gz
else
echo "Debug patch did not trigger (tracing_on still 1)"
fi
Thomas Gleixner
maintainerQuoted 8 lines · Vishal L
Quoted 3 lines · Thomas Gleixner
Thanks for the update. Can you try to provide the information I asked for above?
Ah sorry - I should've said that with your patch applied,
tracing_ondid become 0, so the problem was triggered.The trace from that is in the URL above.
I clearly can't read :)
Quoted 20 lines · Thomas Gleixner
Quoted 19 lines · Vishal L. Verma
I tried out AI assisted and patch (below) which does happen to solve it, but I'm not familiar in this area, and not sure if this is the right fix.
include/linux/entry-virt.h +1
-0@@ -4,6 +4,7 @@#include <linux/static_call_types.h> #include <linux/resume_user_mode.h> +#include <linux/hrtimer_rearm.h> #include <linux/syscalls.h> #include <linux/seccomp.h> #include <linux/sched.h> @@ -58,6 +59,7 @@ int xfer_to_guest_mode_handle_work(void);static inline void xfer_to_guest_mode_prepare (void ) {
Quoted 8 lines
include/linux/entry-virt.h int xfer_to_guest_mode_handle_work(void); · line 65
Quoted 3 lines
include/linux/entry-virt.h +1
-0lockdep_assert_irqs_disabled (); +hrtimer_rearm_deferred ();tick_nohz_user_enter_prepare ();This code should never be reached with a rearm pending. Something else went wrong earlier. So while the patch "works" it papers over the underlying problem.
IIUC, the problem might be:
HRTimer -> VMExit:
[IRQ is disabled]
kvm_x86_call(handle_exit_irqoff)(vcpu)
vmx_handle_exit_irqoff
handle_external_interrupt_irqoff
sysvec_apic_timer_interrupt
irqentry_enter
...
irqentry_exit
irqentry_exit_to_kernel_mode
if (!regs_irqs_disabled(regs)) //<-- This is false, hrtimer
hrtimer_rearm_deferred() rearm is skipped!
This issue is triggered on TDX since TDX can't use preemption timer while normal VMX VM uses preemption timer by default.
Quoted 38 lines
Can you please do the following:
1) Apply the patch below
2) Enable function tracing and the hrtimer* trace events
3) Enable tracing if it has been disabled already
echo 1 >/sys/kernel/tracing/tracing_on
4) Run the tests and wait for
/sys/kernel/tracing/tracing_onto become 0, which means the problem triggered.5) Retrieve the trace from
/sys/kernel/tracing/traceand provide it somewhere to download from or send it to me compressed offlist.Thanks,
tglx
---include/linux/entry-virt.h +4
-0@@ -58,6 +58,10 @@ int xfer_to_guest_mode_handle_work(void);static inline void xfer_to_guest_mode_prepare (void ) {lockdep_assert_irqs_disabled (); +if (test_thread_flag (TIF_HRTIMER_REARM )) { +tracing_off (); +hrtimer_rearm_deferred (); + }tick_nohz_user_enter_prepare (); }
Thomas Gleixner
maintainerQuoted 28 lines · Binbin Wu
Quoted 10 lines · Thomas Gleixner
Quoted 5 lines
static inline void
xfer_to_guest_mode_prepare(void){ lockdep_assert_irqs_disabled(); + hrtimer_rearm_deferred(); tick_nohz_user_enter_prepare();This code should never be reached with a rearm pending. Something else went wrong earlier. So while the patch "works" it papers over the underlying problem.
IIUC, the problem might be:
HRTimer -> VMExit:
[IRQ is disabled]
kvm_x86_call(handle_exit_irqoff)(vcpu)
vmx_handle_exit_irqoff
handle_external_interrupt_irqoff
sysvec_apic_timer_interrupt
irqentry_enter
...
irqentry_exit
irqentry_exit_to_kernel_mode
if (!regs_irqs_disabled(regs)) //<-- This is false, hrtimer
hrtimer_rearm_deferred()rearm is skipped!
Quoted 2 lines
This issue is triggered on TDX since TDX can't use preemption timer while normal VMX VM uses preemption timer by default.
Kinda.
The issue is that vmx_handle_exit_irqoff() always hands in regs with regs->flags.X86_EFLAGS_IF == 0. That has absolutely nothing to do with TDX and the preemption timer.
The patch below solves the problem right there in the exit code, which is unfortunate as there might be a NEED_RESCHED pending. But that can't be taken into account as KVM enables interrupts _before_ reaching the exit work point.
Yet another proof that virt creates more problems than it solves.
Thanks,
tglx
---
Subject: entry: Enforce hrtimer rearming in the irqentry_exit path
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt() invokes hrtimer_rearm_deferred() only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.
So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with pt_regs which have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.
Cure this by adding a hrtimer_rearm_deferred() invocation into the interrupted context has interrupt disabled path of irqentry_exit_to_kernel_mode_after_preempt().
That's unfortunate when there is an actual reschedule pending, but it can't be avoided because KVM invokes a lot of code and also reenables interrupts _before_ reaching the point where the reschedule condition is handled. That can delay the rearming significantly, which in turn can cause artificial latencies.
--- include/linux/irq-entry-common.h | 8 ++++++++ 1 file changed, 8 insertions(+)
include/linux/irq-entry-common.h +8 -0
@@ -516,6 +516,14 @@ irqentry_exit_to_kernel_mode_after_preeminstrumentation_end (); }else {/* +* This is sadly required due to KVM, which invokes regular +* interrupt handlers with interrupt disabled state in @regs. +*/ +instrumentation_begin (); +hrtimer_rearm_deferred (); +instrumentation_end (); + +/* * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry. */
Quoted 11 lines · Thomas Gleixner
---
Subject: entry: Enforce hrtimer rearming in theirqentry_exitpath
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt()invokeshrtimer_rearm_deferred()only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.
Quoted 6 lines
So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.
Mooo :-(
That also complicates the comment that goes with hrtimer_rearm_deferred(). Not sure how to 'fix' that.
Quoted 9 lines
Cure this by adding a
hrtimer_rearm_deferred()invocation into the interrupted context has interrupt disabled path ofirqentry_exit_to_kernel_mode_after_preempt().That's unfortunate when there is an actual reschedule pending, but it can't be avoided because KVM invokes a lot of code and also reenables interrupts _before_ reaching the point where the reschedule condition is handled. That can delay the rearming significantly, which in turn can cause artificial latencies.
Yeah, this is a trainwreck. If they want it better, KVM needs to get 'fixed' to not play silly games like this.
Quoted 4 lines
Fixes 0e98eb14814e ("entry: Prepare for deferred hrtimer rearming")Reported-by "Verma, Vishal L" <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>
Quoted 21 lines
--- include/linux/irq-entry-common.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/linux/irq-entry-common.h +8
-0@@ -516,6 +516,14 @@ irqentry_exit_to_kernel_mode_after_preeminstrumentation_end (); }else {/* +* This is sadly required due to KVM, which invokes regular +* interrupt handlers with interrupt disabled state in @regs. +*/ +instrumentation_begin (); +hrtimer_rearm_deferred (); +instrumentation_end (); + +/* * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry. */
Peter Zijlstra
maintainerQuoted 58 lines · Peter Zijlstra
Quoted 11 lines · Thomas Gleixner
---
Subject: entry: Enforce hrtimer rearming in theirqentry_exitpath
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt()invokeshrtimer_rearm_deferred()only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.Quoted 6 lines
So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.Mooo :-(
That also complicates the comment that goes with
hrtimer_rearm_deferred(). Not sure how to 'fix' that.Quoted 9 lines
Cure this by adding a
hrtimer_rearm_deferred()invocation into the interrupted context has interrupt disabled path ofirqentry_exit_to_kernel_mode_after_preempt().That's unfortunate when there is an actual reschedule pending, but it can't be avoided because KVM invokes a lot of code and also reenables interrupts _before_ reaching the point where the reschedule condition is handled. That can delay the rearming significantly, which in turn can cause artificial latencies.
Yeah, this is a trainwreck. If they want it better, KVM needs to get 'fixed' to not play silly games like this.
Quoted 4 lines
Fixes 0e98eb14814e ("entry: Prepare for deferred hrtimer rearming")Reported-by "Verma, Vishal L" <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>Quoted 13 lines
--- include/linux/irq-entry-common.h | 8 ++++++++ 1 file changed, 8 insertions(+)include/linux/irq-entry-common.h +3
-0@@ -516,6 +516,14 @@ irqentry_exit_to_kernel_mode_after_preeminstrumentation_end (); }else {/* +* This is sadly required due to KVM, which invokes regular +* interrupt handlers with interrupt disabled state in @regs. +*/
Quoted 8 lines
include/linux/irq-entry-common.h irqentry_exit_to_kernel_mode_after_preem · line 529
Quoted 8 lines
include/linux/irq-entry-common.h +5
-0+instrumentation_begin (); +hrtimer_rearm_deferred (); +instrumentation_end (); + +/* * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry. */
Ohhh, wait. What happens if you take a page-fault from NMI context? Does this then not result in trying to program the timer from NMI context?
Peter Zijlstra
maintainerQuoted 14 lines · Peter Zijlstra
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Thomas Gleixner
---
Subject: entry: Enforce hrtimer rearming in theirqentry_exitpath
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt()invokeshrtimer_rearm_deferred()only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all arch/KVM that use GENERIC_ENTRY?
Peter Zijlstra
maintainerQuoted 22 lines · Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Peter Zijlstra
Quoted 17 lines · Thomas Gleixner
---
Subject: entry: Enforce hrtimer rearming in theirqentry_exitpath
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt()invokeshrtimer_rearm_deferred()only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all
arch/KVMthat useGENERIC_ENTRY?
Should we not make asm_fred_entry_from_kvm()/VMX_DO_EVENT_IRQOFF fix IF on the fake frame instead? We know it will enable IRQs after doing handle_exit_irqoff() in vcpu_enter_guest().
SVM does not seem affected with this particular insanity.
Peter Zijlstra
maintainerQuoted 23 lines · Peter Zijlstra
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Peter Zijlstra
Quoted 17 lines · Thomas Gleixner
---
Subject: entry: Enforce hrtimer rearming in theirqentry_exitpath
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt()invokeshrtimer_rearm_deferred()only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all
arch/KVMthat useGENERIC_ENTRY?Should we not make asm_fred_entry_from_kvm()/VMX_DO_EVENT_IRQOFF fix IF on the fake frame instead? We know it will enable IRQs after doing
handle_exit_irqoff()invcpu_enter_guest().
Moo, you can't do that either, because it will ERETS/IRET and fuck up the state :/
Peter Zijlstra
maintainerQuoted 29 lines · Peter Zijlstra
Quoted 28 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Thomas Gleixner
---
Subject: entry: Enforce hrtimer rearming in theirqentry_exitpath
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt()invokeshrtimer_rearm_deferred()only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all
arch/KVMthat useGENERIC_ENTRY?
Quoted 6 lines
Quoted 3 lines
Should we not make asm_fred_entry_from_kvm()/VMX_DO_EVENT_IRQOFF fix IF on the fake frame instead? We know it will enable IRQs after doing
handle_exit_irqoff()invcpu_enter_guest().Moo, you can't do that either, because it will ERETS/IRET and fuck up the state :/
How insane is something like this?
---
arch/x86/entry/entry_64_fred.S +1 -0
@@ -98,6 +98,7 @@ SYM_FUNC_START(asm_fred_entry_from_kvm) push %rdi /* fred_ss handed in by the caller */ push %rbp pushf + or $X86_EFLAGS_KVM, (%rsp) push $__KERNEL_CS /*
arch/x86/include/asm/entry-common.h +12 -0
@@ -97,4 +97,16 @@ static __always_inline void arch_exit_to_user_mode(void) }#define arch_exit_to_user_mode arch_exit_to_user_mode +static __always_inline void arch_exit_to_kernel_mode(struct pt_regs *regs) +{ +#ifdef CONFIG_KVM_INTEL +/* +* KVM is a reserved bit and must always be 0. Hardware will #GP on +* IRET/ERETS with this bit set. +*/ + regs->flags &= ~X86_EFLAGS_KVM ; +#endif +} +#define arch_exit_to_kernel_mode arch_exit_to_kernel_mode +#endif
arch/x86/include/asm/ptrace.h +9 -1
@@ -311,7 +311,15 @@ void user_stack_pointer_set(struct pt_regs *regs, unsigned long val) static __always_inline bool regs_irqs_disabled(struct pt_regs *regs) { -return !(regs->flags &X86_EFLAGS_IF ); +/* +* return context | IF | KVM +* ---------------+----+---- +* IRQ-off | 0 | 0 +* IRQ-on | 0 | 1 +* IRQ-on | 1 | 0 +* invalid | 1 | 1 +*/ +return (regs->flags & (X86_EFLAGS_IF |X86_EFLAGS_KVM )) ==0 ; }/* Query offset/name of register from its name/offset */
arch/x86/include/uapi/asm/processor-flags.h +2 -0
@@ -14,6 +14,8 @@#define X86_EFLAGS_FIXED _BITUL(X86_EFLAGS_FIXED_BIT)#define X86_EFLAGS_PF_BIT 2/* Parity Flag */ #define X86_EFLAGS_PF _BITUL(X86_EFLAGS_PF_BIT) +#define X86_EFLAGS_KVM_BIT 3/* KVM Flag -- must be 0 */ +#define X86_EFLAGS_KVM _BITUL(X86_EFLAGS_PF_BIT)#define X86_EFLAGS_AF_BIT 4/* Auxiliary carry Flag */ #define X86_EFLAGS_AF _BITUL(X86_EFLAGS_AF_BIT)#define X86_EFLAGS_ZF_BIT 6/* Zero Flag */
arch/x86/kvm/vmx/vmenter.S +1 -0
@@ -50,6 +50,7 @@ push %rbp #endif pushf + or $X86_EFLAGS_KVM, (%_ASM_SP) push $__KERNEL_CS \call_insn \call_target
include/linux/irq-entry-common.h +5 -0
@@ -167,6 +167,10 @@ static __always_inline void arch_exit_to_user_mode(void); static __always_inline void arch_exit_to_user_mode(void) { }#endif +#ifndef arch_exit_to_kernel_mode +static __always_inline void arch_exit_to_kernel_mode(struct pt_regs *regs) { } +#endif +/** * arch_do_signal_or_restart - Architecture specific signal delivery function * @regs: Pointer to currents pt_regs
@@ -548,6 +552,7 @@ static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *regs,instrumentation_end ();irqentry_exit_to_kernel_mode_after_preempt (regs, state); +arch_exit_to_kernel_mode (regs); }/**
Peter Zijlstra
maintainerQuoted 30 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Peter Zijlstra
Quoted 26 lines · Peter Zijlstra
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Peter Zijlstra
Quoted 11 lines · Thomas Gleixner
---
Subject: entry: Enforce hrtimer rearming in theirqentry_exitpath
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt()invokeshrtimer_rearm_deferred()only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all
arch/KVMthat useGENERIC_ENTRY?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Should we not make asm_fred_entry_from_kvm()/VMX_DO_EVENT_IRQOFF fix IF on the fake frame instead? We know it will enable IRQs after doing
handle_exit_irqoff()invcpu_enter_guest().Moo, you can't do that either, because it will ERETS/IRET and fuck up the state :/
How insane is something like this?
Small matter of actually building...
---
arch/x86/entry/entry_64_fred.S +1 -0
@@ -98,6 +98,7 @@ SYM_FUNC_START(asm_fred_entry_from_kvm) push %rdi /* fred_ss handed in by the caller */ push %rbp pushf + orq $X86_EFLAGS_KVM, (%rsp) push $__KERNEL_CS /*
arch/x86/include/asm/asm.h +1 -0
@@ -43,6 +43,7 @@#define _ASM_SUB __ASM_SIZE(sub)#define _ASM_XADD __ASM_SIZE(xadd)#define _ASM_MUL __ASM_SIZE(mul) +#define _ASM_OR __ASM_SIZE(or)#define _ASM_AX __ASM_REG(ax)#define _ASM_BX __ASM_REG(bx)
arch/x86/include/asm/entry-common.h +12 -0
@@ -97,4 +97,16 @@ static __always_inline void arch_exit_to_user_mode(void) }#define arch_exit_to_user_mode arch_exit_to_user_mode +static __always_inline void arch_exit_to_kernel_mode(struct pt_regs *regs) +{ +#ifdef CONFIG_KVM_INTEL +/* +* KVM is a reserved bit and must always be 0. Hardware will #GP on +* IRET/ERETS with this bit set. +*/ + regs->flags &= ~X86_EFLAGS_KVM ; +#endif +} +#define arch_exit_to_kernel_mode arch_exit_to_kernel_mode +#endif
arch/x86/include/asm/ptrace.h +9 -1
@@ -311,7 +311,15 @@ void user_stack_pointer_set(struct pt_regs *regs, unsigned long val) static __always_inline bool regs_irqs_disabled(struct pt_regs *regs) { -return !(regs->flags &X86_EFLAGS_IF ); +/* +* return context | IF | KVM +* ---------------+----+---- +* IRQ-off | 0 | 0 +* IRQ-on | 0 | 1 +* IRQ-on | 1 | 0 +* invalid | 1 | 1 +*/ +return (regs->flags & (X86_EFLAGS_IF |X86_EFLAGS_KVM )) ==0 ; }/* Query offset/name of register from its name/offset */
arch/x86/include/uapi/asm/processor-flags.h +2 -0
@@ -14,6 +14,8 @@#define X86_EFLAGS_FIXED _BITUL(X86_EFLAGS_FIXED_BIT)#define X86_EFLAGS_PF_BIT 2/* Parity Flag */ #define X86_EFLAGS_PF _BITUL(X86_EFLAGS_PF_BIT) +#define X86_EFLAGS_KVM_BIT 3/* KVM Flag -- must be 0 */ +#define X86_EFLAGS_KVM _BITUL(X86_EFLAGS_PF_BIT)#define X86_EFLAGS_AF_BIT 4/* Auxiliary carry Flag */ #define X86_EFLAGS_AF _BITUL(X86_EFLAGS_AF_BIT)#define X86_EFLAGS_ZF_BIT 6/* Zero Flag */
arch/x86/kvm/vmx/vmenter.S +2 -0
@@ -6,6 +6,7 @@ #include <asm/nospec-branch.h> #include <asm/percpu.h> #include <asm/segment.h> +#include <asm/processor-flags.h> #include "kvm-asm-offsets.h" #include "run_flags.h"
@@ -50,6 +51,7 @@ push %rbp #endif pushf + _ASM_OR $X86_EFLAGS_KVM, (%_ASM_SP) push $__KERNEL_CS \call_insn \call_target
include/linux/irq-entry-common.h +5 -0
@@ -167,6 +167,10 @@ static __always_inline void arch_exit_to_user_mode(void); static __always_inline void arch_exit_to_user_mode(void) { }#endif +#ifndef arch_exit_to_kernel_mode +static __always_inline void arch_exit_to_kernel_mode(struct pt_regs *regs) { } +#endif +/** * arch_do_signal_or_restart - Architecture specific signal delivery function * @regs: Pointer to currents pt_regs
@@ -548,6 +552,7 @@ static __always_inline void irqentry_exit_to_kernel_mode(struct pt_regs *regs,instrumentation_end ();irqentry_exit_to_kernel_mode_after_preempt (regs, state); +arch_exit_to_kernel_mode (regs); }/**
Thomas Gleixner
maintainerQuoted 5 lines · Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines
Moo, you can't do that either, because it will ERETS/IRET and fuck up the state :/
How insane is something like this?
Pretty insane :)
arch/x86/include/uapi/asm/processor-flags.h line 18
Quoted 12 lines · Peter Zijlstra
arch/x86/include/uapi/asm/processor-flags.h +2
-0@@ -14,6 +14,8 @@ #define X86_EFLAGS_FIXED _BITUL(X86_EFLAGS_FIXED_BIT) #define X86_EFLAGS_PF_BIT 2 /* Parity Flag */ #define X86_EFLAGS_PF _BITUL(X86_EFLAGS_PF_BIT) +#define X86_EFLAGS_KVM_BIT 3/* KVM Flag -- must be 0 */ +#define X86_EFLAGS_KVM _BITUL(X86_EFLAGS_PF_BIT)
I fixed up the copy-paste typo here - _BITUL(X86_EFLAGS_KVM_BIT)
.. and with that the tests pass.
Thomas Gleixner
maintainerQuoted 6 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all
arch/KVMthat useGENERIC_ENTRY?Should we not make asm_fred_entry_from_kvm()/VMX_DO_EVENT_IRQOFF fix IF on the fake frame instead? We know it will enable IRQs after doing
handle_exit_irqoff()invcpu_enter_guest().
Doesn't work :)
Quoted 1 line
SVM does not seem affected with this particular insanity.
Looks like. It will take the interrupt after local_irq_enable().
Now for VMX, that hrtimer_rearm_deferred() call should really go into handle_external_interrupt_irqoff(), which in turn requires to export __hrtimer_rearm_deferred().
But we can avoid that alltogether. Something like the untested below.
Thanks,
tglx
---
kernel/time/hrtimer.c +12 -6
@@ -42,9 +42,10 @@#include <linux/timer.h> #include <linux/freezer.h> #include <linux/compat.h> -#include <linux/uaccess.h> +#include <asm/irq_regs.h> +#include <trace/events/timer.h> #include "tick-internal.h"
@@ -2062,11 +2063,16 @@ void __hrtimer_rearm_deferred(void) static __always_inline voidhrtimer_interrupt_rearm (struct hrtimer_cpu_base *cpu_base,ktime_t expires_next) { -/* hrtimer_interrupt() just re-evaluated the first expiring timer */ - cpu_base->deferred_needs_update = false; -/* Cache the expiry time */ - cpu_base->deferred_expires_next = expires_next; -set_thread_flag (TIF_HRTIMER_REARM ); +/* Lies, damned lies and virt */ +if (likely (!regs_irqs_disabled (get_irq_regs ()))) { +/* hrtimer_interrupt() just re-evaluated the first expiring timer */ + cpu_base->deferred_needs_update = false; +/* Cache the expiry time */ + cpu_base->deferred_expires_next = expires_next; +set_thread_flag (TIF_HRTIMER_REARM ); + }else { +hrtimer_rearm (cpu_base, expires_next, false); + } }#else /* CONFIG_HRTIMER_REARM_DEFERRED */ static __always_inline void
.
Quoted 17 lines · Thomas Gleixner
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all
arch/KVMthat useGENERIC_ENTRY?Should we not make asm_fred_entry_from_kvm()/VMX_DO_EVENT_IRQOFF fix IF on the fake frame instead? We know it will enable IRQs after doing
handle_exit_irqoff()invcpu_enter_guest().
Quoted 5 lines
Doesn't work :)
Quoted 1 line
SVM does not seem affected with this particular insanity.
Looks like. It will take the interrupt after
local_irq_enable().
FWIW, VMX should work just like SVM if we clear VM_EXIT_ACK_INTR_ON_EXIT.
Thomas Gleixner
maintainerQuoted 18 lines · Jim Mattson
Quoted 17 lines · Thomas Gleixner
Quoted 14 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all
arch/KVMthat useGENERIC_ENTRY?Should we not make asm_fred_entry_from_kvm()/VMX_DO_EVENT_IRQOFF fix IF on the fake frame instead? We know it will enable IRQs after doing
handle_exit_irqoff()invcpu_enter_guest().
Quoted 7 lines
Quoted 5 lines
Doesn't work :)
Quoted 1 line
SVM does not seem affected with this particular insanity.
Looks like. It will take the interrupt after
local_irq_enable().FWIW, VMX should work just like SVM if we clear
VM_EXIT_ACK_INTR_ON_EXIT.
I know. What's the point of that VM_EXIT_ACK_INTR_ON_EXIT exercise? Is there any performance benefit or is it just used because it's there?
Thanks,
tglx
Sean Christopherson
maintainerQuoted 19 lines · Thomas Gleixner
Quoted 18 lines · Jim Mattson
Quoted 17 lines · Thomas Gleixner
Quoted 14 lines · Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.Mooo :-(
Also, is this a x86/KVM 'special' or is this true for all
arch/KVMthat useGENERIC_ENTRY?Should we not make asm_fred_entry_from_kvm()/VMX_DO_EVENT_IRQOFF fix IF on the fake frame instead? We know it will enable IRQs after doing
handle_exit_irqoff()invcpu_enter_guest().
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Doesn't work :)
Quoted 1 line
SVM does not seem affected with this particular insanity.
Looks like. It will take the interrupt after
local_irq_enable().FWIW, VMX should work just like SVM if we clear
VM_EXIT_ACK_INTR_ON_EXIT.
Hell no.
Quoted 2 lines
I know. What's the point of that
VM_EXIT_ACK_INTR_ON_EXITexercise? Is there any performance benefit or is it just used because it's there?
There are performance benefits, and it preserves ordering: the first IRQ that's serviced by the host is guaranteed to be _the_ IRQ that triggered the VM-Exit. E.g. with AMD's approach, any IRQs that arrive between the VM-Exit and STI (which is a pretty big swath of code) could be serviced before the IRQ that triggered the exit, depending on priority.
VM_EXIT_ACK_INTR_ON_EXIT also provides symmetry with Intel's handing of NMIs, as NMIs are unconditionally "acked" on VM-Exit.
Even if performance is "fine", changing decades of fundamental KVM behavior is terrifying.
Pulling in an earlier idea:
: Now for VMX, that hrtimer_rearm_deferred() call should really go into : handle_external_interrupt_irqoff(), which in turn requires to export : __hrtimer_rearm_deferred().
IMO, that's the way to go. But instead of exporting __hrtimer_rearm_deferred(), move vmx_do_nmi_irqoff() and vmx_do_interrupt_irqoff() into core kernel entry code (along with the assembly glue), and then EXPORT_SYMBOL_FOR_KVM those. It'd mean some extra surgery, e.g. to provide an equivalent to KVM's IDT lookup:
gate_offset((gate_desc *)host_idt_base + vector)
But I suspect it would be a big net positive in the end.i E.g. the entry code would *know* it's dealing with a direct call from KVM, and thus shouldn't need to play pt_regs games.
Actually, even better would be to bury the FRED vs. not-FRED details in entry code. E.g. on the KVM invocation side, we could get to something like the below, and I'm pretty sure _reduce_ the number of for-KVM exports in the process.
arch/x86/kvm/vmx/vmx.c +3 -14
@@ -7127,17 +7127,9 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu, "unexpected VM-Exit interrupt info: 0x%x", intr_info))return ; -/* -* Invoke the kernel's IRQ handler for the vector. Use the FRED path -* when it's available even if FRED isn't fully enabled, e.g. even if -* FRED isn't supported in hardware, in order to avoid the indirect -* CALL in the non-FRED path. -*/ +/* For the IRQ to the core kernel for processing. */ kvm_before_interrupt (vcpu,KVM_HANDLING_IRQ ); -if (IS_ENABLED (CONFIG_X86_FRED )) -fred_entry_from_kvm (EVENT_TYPE_EXTINT , vector); -else -vmx_do_interrupt_irqoff (gate_offset ((gate_desc *)host_idt_base + vector)); +x86_entry_from_kvm (EVENT_TYPE_EXTINT , vector);kvm_after_interrupt (vcpu); vcpu->arch.at_instruction_boundary = true;
@@ -7447,10 +7439,7 @@ noinstr void vmx_handle_nmi(struct kvm_vcpu *vcpu)return ;kvm_before_interrupt (vcpu,KVM_HANDLING_NMI ); -if (cpu_feature_enabled (X86_FEATURE_FRED )) -fred_entry_from_kvm (EVENT_TYPE_NMI ,NMI_VECTOR ); -else -vmx_do_nmi_irqoff (); +x86_entry_from_kvm (EVENT_TYPE_NMI ,NMI_VECTOR );kvm_after_interrupt (vcpu); }
Peter Zijlstra
maintainerQuoted 10 lines · Sean Christopherson
Pulling in an earlier idea:
: Now for VMX, that
hrtimer_rearm_deferred()call should really go into :handle_external_interrupt_irqoff(), which in turn requires to export :__hrtimer_rearm_deferred().Actually, even better would be to bury the FRED vs. not-FRED details in entry code. E.g. on the KVM invocation side, we could get to something like the below, and I'm pretty sure _reduce_ the number of for-KVM exports in the process.
Something like so then?
arch/x86/entry/Makefile +1 -1
@@ -13,7 +13,7 @@ CFLAGS_REMOVE_syscall_64.o = $(CC_FLAGS_FTRACE) CFLAGS_syscall_32.o += -fno-stack-protector CFLAGS_syscall_64.o += -fno-stack-protector -obj-y := entry.o entry_$(BITS).o syscall_$(BITS).o +obj-y := entry.o entry_$(BITS).o syscall_$(BITS).o common.o obj-y += vdso/ obj-y += vsyscall/
arch/x86/entry/common.c new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <linux/kvm_types.h> +#include <linux/hrtimer_rearm.h> +#include <asm/entry-common.h> +#include <asm/fred.h> +#include <asm/desc.h> + +noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) +{ +#ifdef CONFIG_X86_64 +fred_entry_from_kvm (event_type, vector); +#else +idt_entry_from_kvm (vector); +#endif +if (event_type ==EVENT_TYPE_EXTINT ) { +instrumentation_begin (); +hrtimer_rearm_deferred (); +instrumentation_end (); + } +} +EXPORT_SYMBOL_FOR_KVM (x86_entry_from_kvm);
arch/x86/entry/entry_32.S +33 -0
@@ -1224,3 +1224,36 @@ SYM_CODE_START(rewind_stack_and_make_dead) 1: jmp 1b SYM_CODE_END(rewind_stack_and_make_dead) .popsection + +.pushsection .noinstr.text, "ax" +.macro IDT_DO_EVENT_IRQOFF call_insn call_target + /* + * Unconditionally create a stack frame, getting the correct RSP on the + * stack (for x86-64) would take two instructions anyways, and RBP can + * be used to restore RSP to make objtool happy (see below). + */ + push %ebp + mov %esp, %ebp + + pushf + push $__KERNEL_CS + \call_insn \call_target + + /* + * "Restore" RSP from RBP, even though IRET has already unwound RSP to + * the correct value. objtool doesn't know the callee will IRET and, + * without the explicit restore, thinks the stack is getting walloped. + * Using an unwind hint is problematic due to x86-64's dynamic alignment. + */ + leave + RET +.endm + +SYM_FUNC_START(idt_do_interrupt_irqoff) + IDT_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 +SYM_FUNC_END(idt_do_interrupt_irqoff) + +SYM_FUNC_START(idt_do_nmi_irqoff) + IDT_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx +SYM_FUNC_END(idt_do_nmi_irqoff) +.popsection
arch/x86/entry/entry_64_fred.S +0 -1
@@ -147,5 +147,4 @@ SYM_FUNC_START(asm_fred_entry_from_kvm) RET SYM_FUNC_END(asm_fred_entry_from_kvm) -EXPORT_SYMBOL_FOR_KVM(asm_fred_entry_from_kvm); #endif
arch/x86/include/asm/desc.h +1 -0
@@ -437,6 +437,7 @@ extern void idt_setup_early_traps(void);extern void idt_setup_traps (void );extern void idt_setup_apic_and_irq_gates (void );extern bool idt_is_f00f_address (unsigned long address); +extern void idt_entry_from_kvm (unsigned int vector);#ifdef CONFIG_X86_64 extern void idt_setup_early_pf (void );
arch/x86/include/asm/entry-common.h +2 -0
@@ -97,4 +97,6 @@ static __always_inline void arch_exit_to_user_mode(void) }#define arch_exit_to_user_mode arch_exit_to_user_mode +extern void x86_entry_from_kvm (unsigned int entry_type,unsigned int vector); +#endif
arch/x86/kernel/idt.c +8 -0
@@ -266,6 +266,14 @@ void __init idt_setup_early_pf(void)idt_setup_from_table (idt_table, early_pf_idts, ARRAY_SIZE(early_pf_idts), true); } +#else +void idt_entry_from_kvm (unsigned int vector) +{ +if (vector ==NMI_VECTOR ) +idt_do_nmi_irqoff (); +else +idt_do_interrupt_irqoff (gate_offset (idt_table + vector)); +}#endif static void __init idt_map_in_cea (void )
arch/x86/kvm/vmx/vmenter.S +0 -46
61 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +3 -14
@@ -7127,17 +7127,9 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu, "unexpected VM-Exit interrupt info: 0x%x", intr_info)) return; -/* -* Invoke the kernel's IRQ handler for the vector. Use the FRED path -* when it's available even if FRED isn't fully enabled, e.g. even if -* FRED isn't supported in hardware, in order to avoid the indirect -* CALL in the non-FRED path. -*/ +/* For the IRQ to the core kernel for processing. */ kvm_before_interrupt (vcpu,KVM_HANDLING_IRQ ); -if (IS_ENABLED (CONFIG_X86_FRED )) -fred_entry_from_kvm (EVENT_TYPE_EXTINT , vector); -else -vmx_do_interrupt_irqoff (gate_offset ((gate_desc *)host_idt_base + vector)); +x86_entry_from_kvm (EVENT_TYPE_EXTINT , vector);kvm_after_interrupt (vcpu); vcpu->arch.at_instruction_boundary = true;
@@ -7447,10 +7439,7 @@ noinstr void vmx_handle_nmi(struct kvm_vcpu *vcpu)return ;kvm_before_interrupt (vcpu,KVM_HANDLING_NMI ); -if (cpu_feature_enabled (X86_FEATURE_FRED )) -fred_entry_from_kvm (EVENT_TYPE_NMI ,NMI_VECTOR ); -else -vmx_do_nmi_irqoff (); +x86_entry_from_kvm (EVENT_TYPE_NMI ,NMI_VECTOR );kvm_after_interrupt (vcpu); }
Peter Zijlstra
maintainerQuoted 9 lines · Peter Zijlstra
Quoted 6 lines · Sean Christopherson
Pulling in an earlier idea:
: Now for VMX, that
hrtimer_rearm_deferred()call should really go into :handle_external_interrupt_irqoff(), which in turn requires to export :__hrtimer_rearm_deferred().
Quoted 5 lines
Quoted 3 lines
Actually, even better would be to bury the FRED vs. not-FRED details in entry code. E.g. on the KVM invocation side, we could get to something like the below, and I'm pretty sure _reduce_ the number of for-KVM exports in the process.
Something like so then?
And this one seems to build on ARCH=i386 too.
---
arch/x86/entry/Makefile +1 -1
@@ -13,7 +13,7 @@ CFLAGS_REMOVE_syscall_64.o = $(CC_FLAGS_FTRACE) CFLAGS_syscall_32.o += -fno-stack-protector CFLAGS_syscall_64.o += -fno-stack-protector -obj-y := entry.o entry_$(BITS).o syscall_$(BITS).o +obj-y := entry.o entry_$(BITS).o syscall_$(BITS).o common.o obj-y += vdso/ obj-y += vsyscall/
arch/x86/entry/common.c new file +22 -0
@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <linux/entry-common.h> +#include <linux/kvm_types.h> +#include <linux/hrtimer_rearm.h> +#include <asm/fred.h> +#include <asm/desc.h> + +noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) +{ +#ifdef CONFIG_X86_64 +fred_entry_from_kvm (event_type, vector); +#else +idt_entry_from_kvm (vector); +#endif +if (event_type ==EVENT_TYPE_EXTINT ) { +instrumentation_begin (); +hrtimer_rearm_deferred (); +instrumentation_end (); + } +} +EXPORT_SYMBOL_FOR_KVM (x86_entry_from_kvm);
arch/x86/entry/entry_32.S +33 -0
@@ -1224,3 +1224,36 @@ SYM_CODE_START(rewind_stack_and_make_dead) 1: jmp 1b SYM_CODE_END(rewind_stack_and_make_dead) .popsection + +.pushsection .noinstr.text, "ax" +.macro IDT_DO_EVENT_IRQOFF call_insn call_target + /* + * Unconditionally create a stack frame, getting the correct RSP on the + * stack (for x86-64) would take two instructions anyways, and RBP can + * be used to restore RSP to make objtool happy (see below). + */ + push %ebp + mov %esp, %ebp + + pushf + push $__KERNEL_CS + \call_insn \call_target + + /* + * "Restore" RSP from RBP, even though IRET has already unwound RSP to + * the correct value. objtool doesn't know the callee will IRET and, + * without the explicit restore, thinks the stack is getting walloped. + * Using an unwind hint is problematic due to x86-64's dynamic alignment. + */ + leave + RET +.endm + +SYM_FUNC_START(idt_do_interrupt_irqoff) + IDT_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 +SYM_FUNC_END(idt_do_interrupt_irqoff) + +SYM_FUNC_START(idt_do_nmi_irqoff) + IDT_DO_EVENT_IRQOFF call asm_exc_nmi +SYM_FUNC_END(idt_do_nmi_irqoff) +.popsection
arch/x86/entry/entry_64_fred.S +0 -1
@@ -147,5 +147,4 @@ SYM_FUNC_START(asm_fred_entry_from_kvm) RET SYM_FUNC_END(asm_fred_entry_from_kvm) -EXPORT_SYMBOL_FOR_KVM(asm_fred_entry_from_kvm); #endif
arch/x86/include/asm/desc.h +4 -0
@@ -438,6 +438,10 @@ extern void idt_setup_traps(void);extern void idt_setup_apic_and_irq_gates (void );extern bool idt_is_f00f_address (unsigned long address); +extern void idt_do_interrupt_irqoff (unsigned int vector); +extern void idt_do_nmi_irqoff (void ); +extern void idt_entry_from_kvm (unsigned int vector); +#ifdef CONFIG_X86_64 extern void idt_setup_early_pf (void );#else
arch/x86/include/asm/entry-common.h +2 -0
@@ -97,4 +97,6 @@ static __always_inline void arch_exit_to_user_mode(void) }#define arch_exit_to_user_mode arch_exit_to_user_mode +extern void x86_entry_from_kvm (unsigned int entry_type,unsigned int vector); +#endif
arch/x86/include/asm/idtentry.h +0 -11
@@ -633,17 +633,6 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_MC, xenpv_exc_machine_check);#endif /* NMI */ - -#if IS_ENABLED (CONFIG_KVM_INTEL ) -/* -* Special entry point for VMX which invokes this on the kernel stack, even for -* 64-bit, i.e. without using an IST. asm_exc_nmi() requires an IST to work -* correctly vs. the NMI 'executing' marker. Used for 32-bit kernels as well -* to avoid more ifdeffery. -*/ -DECLARE_IDTENTRY (X86_TRAP_NMI , exc_nmi_kvm_vmx); -#endif -DECLARE_IDTENTRY_NMI (X86_TRAP_NMI , exc_nmi);#ifdef CONFIG_XEN_PV DECLARE_IDTENTRY_RAW (X86_TRAP_NMI , xenpv_exc_nmi);
arch/x86/kernel/idt.c +8 -0
@@ -266,6 +266,14 @@ void __init idt_setup_early_pf(void)idt_setup_from_table (idt_table, early_pf_idts, ARRAY_SIZE(early_pf_idts), true); } +#else +void idt_entry_from_kvm (unsigned int vector) +{ +if (vector ==NMI_VECTOR ) +idt_do_nmi_irqoff (); +else +idt_do_interrupt_irqoff (gate_offset (idt_table + vector)); +}#endif static void __init idt_map_in_cea (void )
arch/x86/kernel/nmi.c +0 -8
@@ -609,14 +609,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi) goto nmi_restart; } -#if IS_ENABLED (CONFIG_KVM_INTEL ) -DEFINE_IDTENTRY_RAW (exc_nmi_kvm_vmx) -{ -exc_nmi (regs); -} -EXPORT_SYMBOL_FOR_KVM (asm_exc_nmi_kvm_vmx); -#endif -#ifdef CONFIG_NMI_CHECK_CPU static char *nmi_check_stall_msg[] = {
arch/x86/kvm/vmx/vmenter.S +0 -46
61 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +3 -14
@@ -7127,17 +7127,9 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu, "unexpected VM-Exit interrupt info: 0x%x", intr_info)) return; -/* -* Invoke the kernel's IRQ handler for the vector. Use the FRED path -* when it's available even if FRED isn't fully enabled, e.g. even if -* FRED isn't supported in hardware, in order to avoid the indirect -* CALL in the non-FRED path. -*/ +/* For the IRQ to the core kernel for processing. */ kvm_before_interrupt (vcpu,KVM_HANDLING_IRQ ); -if (IS_ENABLED (CONFIG_X86_FRED )) -fred_entry_from_kvm (EVENT_TYPE_EXTINT , vector); -else -vmx_do_interrupt_irqoff (gate_offset ((gate_desc *)host_idt_base + vector)); +x86_entry_from_kvm (EVENT_TYPE_EXTINT , vector);kvm_after_interrupt (vcpu); vcpu->arch.at_instruction_boundary = true;
@@ -7447,10 +7439,7 @@ noinstr void vmx_handle_nmi(struct kvm_vcpu *vcpu)return ;kvm_before_interrupt (vcpu,KVM_HANDLING_NMI ); -if (cpu_feature_enabled (X86_FEATURE_FRED )) -fred_entry_from_kvm (EVENT_TYPE_NMI ,NMI_VECTOR ); -else -vmx_do_nmi_irqoff (); +x86_entry_from_kvm (EVENT_TYPE_NMI ,NMI_VECTOR );kvm_after_interrupt (vcpu); }
Sean Christopherson
maintainerQuoted 9 lines · Peter Zijlstra
Quoted 6 lines · Sean Christopherson
Pulling in an earlier idea:
: Now for VMX, that
hrtimer_rearm_deferred()call should really go into :handle_external_interrupt_irqoff(), which in turn requires to export :__hrtimer_rearm_deferred().
Quoted 5 lines
Quoted 3 lines
Actually, even better would be to bury the FRED vs. not-FRED details in entry code. E.g. on the KVM invocation side, we could get to something like the below, and I'm pretty sure _reduce_ the number of for-KVM exports in the process.
Something like so then?
Yep!
arch/x86/entry/common.c line 4
Quoted 10 lines
arch/x86/entry/common.c new file +4
-0@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <linux/kvm_types.h> +#include <linux/hrtimer_rearm.h>
For CONFIG_X86_FRED=n, which is possible on x86-64 if CONFIG_KVM_INTEL=n, this
#include <linux/sched/task_stack.h>
is needed so that task_pt_regs() can find task_stack_page() (and including task_stack.h in processor.h would create cyclical includes).
arch/x86/entry/common.c line 8
Quoted 4 lines
arch/x86/entry/common.c new file +4
-0+#include <asm/entry-common.h> +#include <asm/fred.h> +#include <asm/desc.h> +
Related to CONFIG_X86_FRED=n, I vote to wrap this API with #if IS_ENABLED(CONFIG_KVM_INTEL) and then delete the fred_entry_from_kvm() stub so that a goof results in a build failure. That'd also be a good place for a comment to explain some of the usage.
Quoted 7 lines
arch/x86/entry/common.c new file +7
-0+noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) +{ +#ifdef CONFIG_X86_64 +fred_entry_from_kvm (event_type, vector); +#else +idt_entry_from_kvm (vector); +#endif
...
arch/x86/entry/common.c
Quoted 7 lines
arch/x86/entry/common.c new file +7
-0+SYM_FUNC_START (idt_do_interrupt_irqoff) + IDT_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 +SYM_FUNC_END(idt_do_interrupt_irqoff) + +SYM_FUNC_START(idt_do_nmi_irqoff) + IDT_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx +SYM_FUNC_END(idt_do_nmi_irqoff)
These need to be declared, and the KVM declarations can be deleted.
Quoted 17 lines
static void
__initidt_map_in_cea(void)arch/x86/kvm/vmx/vmenter.S +0
-8@@ -31,38 +31,6 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif -.macro VMX_DO_EVENT_IRQOFF call_insn call_target - /* - * Unconditionally create a stack frame, getting the correct RSP on the - * stack (for x86-64) would take two instructions anyways, and RBP can - * be used to restore RSP to make objtool happy (see below). - */ - push %_ASM_BP - mov %_ASM_SP, %_ASM_BP
arch/x86/kvm/vmx/vmenter.S line 51
Quoted 10 lines
arch/x86/kvm/vmx/vmenter.S +0
-10- -#ifdef CONFIG_X86_64 - /* - * Align RSP to a 16-byte boundary (to emulate CPU behavior) before - * creating the synthetic interrupt stack frame for the IRQ/NMI. - */ - and $-16, %rsp - push $__KERNEL_DS - push %rbp -#endif
For anyone else having an -ENOCOFFEE moment, this has been dead code since commit 28d11e4548b7 ("x86/fred: KVM: VMX: Always use FRED for IRQs when CONFIG_X86_FRED=y").
This as delta? (I had typed this all up before Peter posted a new verison, so dammit I'm sending it!)
arch/x86/entry/common.c +11 -0
@@ -2,10 +2,20 @@#include <linux/kvm_types.h> #include <linux/hrtimer_rearm.h> +#include <linux/sched/task_stack.h> #include <asm/entry-common.h> #include <asm/fred.h> #include <asm/desc.h> +#if IS_ENABLED (CONFIG_KVM_INTEL ) +/* +* On VMX, NMIs and IRQs (as configured by KVM) are acknowledge by hardware as +* part of the VM-Exit, i.e. the event itself is consumed as part the VM-Exit. +* x86_entry_from_kvm() is invoked by KVM to effectively forward NMIs and IRQs +* to the kernel for servicing. On SVM, a.k.a. AMD, the NMI/IRQ VM-Exit is +* purely a signal that an NMI/IRQ is pending, i.e. the event that triggered +* the VM-Exit is held pending until it's unblocked in the host. +*/ noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) {#ifdef CONFIG_X86_64
@@ -20,3 +30,4 @@ noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) } }EXPORT_SYMBOL_FOR_KVM (x86_entry_from_kvm); +#endif
arch/x86/include/asm/entry-common.h +2 -0
@@ -98,5 +98,7 @@ static __always_inline void arch_exit_to_user_mode(void)#define arch_exit_to_user_mode arch_exit_to_user_modeextern void x86_entry_from_kvm (unsigned int entry_type,unsigned int vector); +extern void idt_do_interrupt_irqoff (unsigned long entry); +extern void idt_do_nmi_irqoff (void );#endif
arch/x86/include/asm/fred.h +0 -1
@@ -110,7 +110,6 @@ static __always_inline unsigned long fred_event_data(struct pt_regs *regs) { retstatic inline void cpu_init_fred_exceptions (void ) { }static inline void cpu_init_fred_rsps (void ) { }static inline void fred_complete_exception_setup (void ) { } -static inline void fred_entry_from_kvm (unsigned int type,unsigned int vector) { }static inline void fred_sync_rsp0 (unsigned long rsp0) { }static inline void fred_update_rsp0 (void ) { }#endif /* CONFIG_X86_FRED */
arch/x86/kernel/nmi.c +0 -1
@@ -614,7 +614,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi_kvm_vmx) {exc_nmi (regs); } -EXPORT_SYMBOL_FOR_KVM (asm_exc_nmi_kvm_vmx);#endif #ifdef CONFIG_NMI_CHECK_CPU
arch/x86/kvm/vmx/vmx.c +0 -3
@@ -7083,9 +7083,6 @@ void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap)vmcs_write64 (EOI_EXIT_BITMAP3 , eoi_exit_bitmap[3 ]); } -void vmx_do_interrupt_irqoff (unsigned long entry); -void vmx_do_nmi_irqoff (void ); -static void handle_nm_fault_irqoff (struct kvm_vcpu *vcpu) {/*
Peter Zijlstra
maintainerQuoted 13 lines · Sean Christopherson
arch/x86/entry/common.c line 4
Quoted 10 lines
arch/x86/entry/common.c new file +4
-0@@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <linux/kvm_types.h> +#include <linux/hrtimer_rearm.h> For
CONFIG_X86_FRED=n, which is possible on x86-64 ifCONFIG_KVM_INTEL=n, this
Quoted 6 lines
#include <
linux/sched/task_stack.h>is needed so that
task_pt_regs()can findtask_stack_page()(and includingtask_stack.hinprocessor.hwould create cyclical includes).Quoted 1 line
arch/x86/entry/common.c new file +1
-0+#include <asm/entry-common.h>
This wanted to be linux/entry-common.h, otherwise I could not get 32bit to build. And that sorts that same include issue you pointed out. My x86_64 build didn't seem to care...
Quoted 7 lines
arch/x86/entry/common.c line 8
Quoted 3 lines
arch/x86/entry/common.c new file +3
-0+#include <asm/fred.h> +#include <asm/desc.h> +Related to
CONFIG_X86_FRED=n, I vote to wrap this API with #ifIS_ENABLED(CONFIG_KVM_INTEL)and then delete thefred_entry_from_kvm()stub so that a goof results in a build failure. That'd also be a good place for a comment to explain some of the usage.
Yeah, it definitely wants a few comments. I'll do the KVM_INTEL thing, I'd forgotten we'd made the FRED=y thing conditinoal on that, I thought we'd had it unconditionally =y for 64bit.
Quoted 2 lines
This as delta? (I had typed this all up before Peter posted a new verison, so dammit I'm sending it!)
:-)
I'll go stare at it in the morning, I'm about to go crash out.
Sean Christopherson
maintainerQuoted 7 lines · Peter Zijlstra
Quoted 2 lines · Sean Christopherson
This as delta? (I had typed this all up before Peter posted a new verison, so dammit I'm sending it!)
:-)
I'll go stare at it in the morning, I'm about to go crash out.
New delta against your effective v2, builds all of my configs (which isn't _that_ many, but I think they cover most of the weirder ways to include KVM (or not)).
I'll start testing the full thing to try and get early signal on the health.
x86/entry/common.c +10 -0
@@ -6,6 +6,15 @@#include <asm/fred.h> #include <asm/desc.h> +#if IS_ENABLED (CONFIG_KVM_INTEL ) +/* +* On VMX, NMIs and IRQs (as configured by KVM) are acknowledge by hardware as +* part of the VM-Exit, i.e. the event itself is consumed as part the VM-Exit. +* x86_entry_from_kvm() is invoked by KVM to effectively forward NMIs and IRQs +* to the kernel for servicing. On SVM, a.k.a. AMD, the NMI/IRQ VM-Exit is +* purely a signal that an NMI/IRQ is pending, i.e. the event that triggered +* the VM-Exit is held pending until it's unblocked in the host. +*/ noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) {#ifdef CONFIG_X86_64
@@ -20,3 +29,4 @@ noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) } }EXPORT_SYMBOL_FOR_KVM (x86_entry_from_kvm); +#endif
x86/include/asm/fred.h +0 -1
@@ -110,7 +110,6 @@ static __always_inline unsigned long fred_event_data(struct pt_regs *regs) { retstatic inline void cpu_init_fred_exceptions (void ) { }static inline void cpu_init_fred_rsps (void ) { }static inline void fred_complete_exception_setup (void ) { } -static inline void fred_entry_from_kvm (unsigned int type,unsigned int vector) { }static inline void fred_sync_rsp0 (unsigned long rsp0) { }static inline void fred_update_rsp0 (void ) { }#endif /* CONFIG_X86_FRED */
x86/kernel/idt.c +1 -1
@@ -267,7 +267,7 @@ void __init idt_setup_early_pf(void) ARRAY_SIZE(early_pf_idts), true); }#else -void idt_entry_from_kvm (unsigned int vector) +noinstr void idt_entry_from_kvm(unsigned int vector) {if (vector ==NMI_VECTOR )idt_do_nmi_irqoff ();
x86/kvm/vmx/vmx.c +0 -3
@@ -7083,9 +7083,6 @@ void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap)vmcs_write64 (EOI_EXIT_BITMAP3 , eoi_exit_bitmap[3 ]); } -void vmx_do_interrupt_irqoff (unsigned long entry); -void vmx_do_nmi_irqoff (void ); -static void handle_nm_fault_irqoff (struct kvm_vcpu *vcpu) {/*
Peter Zijlstra
maintainerQuoted 5 lines · Sean Christopherson
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Sean Christopherson
This as delta? (I had typed this all up before Peter posted a new verison, so dammit I'm sending it!)
Quoted 8 lines
Quoted 3 lines
:-)
I'll go stare at it in the morning, I'm about to go crash out.
New delta against your effective v2, builds all of my configs (which isn't _that_ many, but I think they cover most of the weirder ways to include KVM (or not)).
I'll start testing the full thing to try and get early signal on the health.
Oh, re-reading commit 28d11e4548b7, I think this wrecks NMIs. The patch will also use the FRED NMI path and that's not good when running IDT.
I'll go fix that and stick in a few more comments to clarify this magic.
Peter Zijlstra
maintainerQuoted 13 lines · Peter Zijlstra
Quoted 12 lines · Sean Christopherson
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Sean Christopherson
This as delta? (I had typed this all up before Peter posted a new verison, so dammit I'm sending it!)
:-)
I'll go stare at it in the morning, I'm about to go crash out.
New delta against your effective v2, builds all of my configs (which isn't _that_ many, but I think they cover most of the weirder ways to include KVM (or not)).
Quoted 6 lines
Quoted 1 line
I'll start testing the full thing to try and get early signal on the health.
Oh, re-reading commit
28d11e4548b7, I think this wrecks NMIs. The patch will also use the FRED NMI path and that's not good when running IDT.I'll go fix that and stick in a few more comments to clarify this magic.
This should probably we at least two patches, one moving the code into the x86 core and one adding that hrtimer fix on top. But kept as one for now.
Irrespective of the hrtimer fix, I think the initial move into x86 core part is a sane move.
---
Subject: x86/kvm/vmx: Move IRQ/NMI dispatch from KVM into x86
Move the VMX interrupt dispatch magic into the x86 core code. This isolates KVM from the FRED/IDT decisions and reduces the amount of EXPORT_SYMBOL_FOR_KVM().
---
arch/x86/entry/Makefile +1 -1
@@ -13,7 +13,7 @@ CFLAGS_REMOVE_syscall_64.o = $(CC_FLAGS_FTRACE) CFLAGS_syscall_32.o += -fno-stack-protector CFLAGS_syscall_64.o += -fno-stack-protector -obj-y := entry.o entry_$(BITS).o syscall_$(BITS).o +obj-y := entry.o entry_$(BITS).o syscall_$(BITS).o common.o obj-y += vdso/ obj-y += vsyscall/
arch/x86/entry/common.c new file +62 -0
62 lines not on this page · raw message
arch/x86/entry/entry.S +46 -0
49 lines not on this page · raw message
arch/x86/entry/entry_64_fred.S +0 -1
@@ -147,5 +147,4 @@ SYM_FUNC_START(asm_fred_entry_from_kvm) RET SYM_FUNC_END(asm_fred_entry_from_kvm) -EXPORT_SYMBOL_FOR_KVM(asm_fred_entry_from_kvm); #endif
arch/x86/include/asm/desc.h +4 -0
@@ -438,6 +438,10 @@ extern void idt_setup_traps(void);extern void idt_setup_apic_and_irq_gates (void );extern bool idt_is_f00f_address (unsigned long address); +extern void idt_do_interrupt_irqoff (unsigned int vector); +extern void idt_do_nmi_irqoff (void ); +extern void idt_entry_from_kvm (unsigned int vector); +#ifdef CONFIG_X86_64 extern void idt_setup_early_pf (void );#else
arch/x86/include/asm/desc_defs.h +1 -1
@@ -145,7 +145,7 @@ struct gate_struct {typedef struct gate_struct gate_desc ;#ifndef _SETUP -static inline unsigned long gate_offset (const gate_desc *g) +static __always_inline unsigned long gate_offset (const gate_desc *g) {#ifdef CONFIG_X86_64 return g->offset_low | ((unsigned long)g->offset_middle << 16) |
arch/x86/include/asm/entry-common.h +2 -0
@@ -97,4 +97,6 @@ static __always_inline void arch_exit_to_user_mode(void) }#define arch_exit_to_user_mode arch_exit_to_user_mode +extern void x86_entry_from_kvm (unsigned int entry_type,unsigned int vector); +#endif
arch/x86/include/asm/fred.h +0 -1
@@ -110,7 +110,6 @@ static __always_inline unsigned long fred_event_data(struct pt_regs *regs) { retstatic inline void cpu_init_fred_exceptions (void ) { }static inline void cpu_init_fred_rsps (void ) { }static inline void fred_complete_exception_setup (void ) { } -static inline void fred_entry_from_kvm (unsigned int type,unsigned int vector) { }static inline void fred_sync_rsp0 (unsigned long rsp0) { }static inline void fred_update_rsp0 (void ) { }#endif /* CONFIG_X86_FRED */
arch/x86/include/asm/idtentry.h +0 -11
@@ -633,17 +633,6 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_MC, xenpv_exc_machine_check);#endif /* NMI */ - -#if IS_ENABLED (CONFIG_KVM_INTEL ) -/* -* Special entry point for VMX which invokes this on the kernel stack, even for -* 64-bit, i.e. without using an IST. asm_exc_nmi() requires an IST to work -* correctly vs. the NMI 'executing' marker. Used for 32-bit kernels as well -* to avoid more ifdeffery. -*/ -DECLARE_IDTENTRY (X86_TRAP_NMI , exc_nmi_kvm_vmx); -#endif -DECLARE_IDTENTRY_NMI (X86_TRAP_NMI , exc_nmi);#ifdef CONFIG_XEN_PV DECLARE_IDTENTRY_RAW (X86_TRAP_NMI , xenpv_exc_nmi);
arch/x86/kernel/idt.c +13 -0
@@ -268,6 +268,19 @@ void __init idt_setup_early_pf(void) }#endif +noinstr void idt_entry_from_kvm(unsigned int vector) +{ +if (vector ==NMI_VECTOR ) +return idt_do_nmi_irqoff (); + +/* +* Only the NMI path requires noinstr. +*/ +instrumentation_begin (); +idt_do_interrupt_irqoff (gate_offset (idt_table + vector)); +instrumentation_end (); +} +static void __init idt_map_in_cea (void ) {/*
arch/x86/kernel/nmi.c +0 -8
@@ -609,14 +609,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi) goto nmi_restart; } -#if IS_ENABLED (CONFIG_KVM_INTEL ) -DEFINE_IDTENTRY_RAW (exc_nmi_kvm_vmx) -{ -exc_nmi (regs); -} -EXPORT_SYMBOL_FOR_KVM (asm_exc_nmi_kvm_vmx); -#endif -#ifdef CONFIG_NMI_CHECK_CPU static char *nmi_check_stall_msg[] = {
arch/x86/kvm/vmx/vmenter.S +0 -46
61 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +3 -17
@@ -7083,9 +7083,6 @@ void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap)vmcs_write64 (EOI_EXIT_BITMAP3 , eoi_exit_bitmap[3 ]); } -void vmx_do_interrupt_irqoff (unsigned long entry); -void vmx_do_nmi_irqoff (void ); -static void handle_nm_fault_irqoff (struct kvm_vcpu *vcpu) {/*
@@ -7127,17 +7124,9 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu, "unexpected VM-Exit interrupt info: 0x%x", intr_info)) return; -/* -* Invoke the kernel's IRQ handler for the vector. Use the FRED path -* when it's available even if FRED isn't fully enabled, e.g. even if -* FRED isn't supported in hardware, in order to avoid the indirect -* CALL in the non-FRED path. -*/ +/* For the IRQ to the core kernel for processing. */ kvm_before_interrupt (vcpu,KVM_HANDLING_IRQ ); -if (IS_ENABLED (CONFIG_X86_FRED )) -fred_entry_from_kvm (EVENT_TYPE_EXTINT , vector); -else -vmx_do_interrupt_irqoff (gate_offset ((gate_desc *)host_idt_base + vector)); +x86_entry_from_kvm (EVENT_TYPE_EXTINT , vector);kvm_after_interrupt (vcpu); vcpu->arch.at_instruction_boundary = true;
@@ -7447,10 +7436,7 @@ noinstr void vmx_handle_nmi(struct kvm_vcpu *vcpu)return ;kvm_before_interrupt (vcpu,KVM_HANDLING_NMI ); -if (cpu_feature_enabled (X86_FEATURE_FRED )) -fred_entry_from_kvm (EVENT_TYPE_NMI ,NMI_VECTOR ); -else -vmx_do_nmi_irqoff (); +x86_entry_from_kvm (EVENT_TYPE_NMI ,NMI_VECTOR );kvm_after_interrupt (vcpu); }
Peter Zijlstra
maintainerarch/x86/entry/common.c line 42
Quoted 9 lines · Peter Zijlstra
arch/x86/entry/common.c new file +9
-0+instrumentation_begin (); +/* +* KVM/VMX will dispatch from IRQ-disabled but for a context +* that will have IRQs-enabled. This confuses the entry code +* and it will not have reprogrammed the timer (or do +* preemption). Minimal fixup for now. +*/ +hrtimer_rearm_deferred (); +instrumentation_end ();
So I've been looking at this preemption thing. After having gotten my head in a twist a few times around, I think this is done by vcpu_enter_guest() like:
preempt_disable(); local_irq_disable(); ... kvm_x86_call(handle_exit_irqoff)(vcpu); <--- all of this VMX nonsense ... local_irq_enable(); <--- WTF goes here :-) ++vcpu->stat.exits; local_irq_disable(); ... local_irq_enable(); preempt_enable(); <--- here we finally preempt
This earlier IRQ-enable makes my head hurt and I had to go buy a new WTF'o'meter (again!).
Sean Christopherson
maintainerQuoted 15 lines · Peter Zijlstra
arch/x86/entry/common.c line 42
Quoted 9 lines · Peter Zijlstra
arch/x86/entry/common.c new file +9
-0+instrumentation_begin (); +/* +* KVM/VMX will dispatch from IRQ-disabled but for a context +* that will have IRQs-enabled. This confuses the entry code +* and it will not have reprogrammed the timer (or do +* preemption). Minimal fixup for now. +*/ +hrtimer_rearm_deferred (); +instrumentation_end ();So I've been looking at this preemption thing. After having gotten my head in a twist a few times around, I think this is done by
vcpu_enter_guest()like:
Quoted 4 lines
preempt_disable(); local_irq_disable(); ... kvm_x86_call(handle_exit_irqoff)(vcpu); <--- all of this VMX nonsense
To be fair, there's some SVM nonsense too.
Quoted 3 lines
... local_irq_enable(); <--- WTF goes here :-)
All IRQs on AMD, and tick IRQs on VMX that arrive _just_ after the VM-Exit.
Because IRQ exits on AMD/SVM are purely a notification, KVM needs to enable IRQs in order to service the exit. The early enabling exists to get the timeslice accounting correct. With the comments...
/* * Consume any pending interrupts, including the possible source of * VM-Exit on SVM and any ticks that occur between VM-Exit and now. * An instruction is required after local_irq_enable() to fully unblock * interrupts on processors that implement an interrupt shadow, the * stat.exits increment will do nicely. */ kvm_before_interrupt(vcpu, KVM_HANDLING_IRQ); local_irq_enable(); ++vcpu->stat.exits; local_irq_disable(); kvm_after_interrupt(vcpu); /* * Wait until after servicing IRQs to account guest time so that any * ticks that occurred while running the guest are properly accounted * to the guest. Waiting until IRQs are enabled degrades the accuracy * of accounting via context tracking, but the loss of accuracy is * acceptable for all known use cases. */ guest_timing_exit_irqoff();
When NOT precisely accounting virtual CPU usage, accounting is done by setting PF_VCPU on current->flags on the way into the guest, and then clearing it on the way back out. The latter is done by guest_timing_exit_irqoff(). If KVM waits to enable IRQs until after guest_timing_exit_irqoff(), then the tick IRQ handler will account the tick to the host, not that guest, even if 99.99999% of the time was spent in the guest.
As to why this is in common code, i.e. isn't AMD/SVM specific, if the guest and host are running with the same tick frequency, it's suprisingly easy to get in a state where the host tick IRQ almost always arrives just after VM-Exit, before KVM fully enables IRQs. Specifically, the guest's programmed tick will trigger a VMX Preemption Timer VM-Exit at the same frequency the host's tick triggers an IRQ.
Quoted 8 lines
++vcpu->stat.exits; local_irq_disable(); ... local_irq_enable(); preempt_enable(); <--- here we finally preemptThis earlier IRQ-enable makes my head hurt and I had to go buy a new WTF'o'meter (again!).
Quoted 12 lines · Peter Zijlstra
This should probably we at least two patches, one moving the code into the x86 core and one adding that hrtimer fix on top. But kept as one for now.
Irrespective of the hrtimer fix, I think the initial move into x86 core part is a sane move.
---
Subject: x86/kvm/vmx: Move IRQ/NMI dispatch from KVM into x86
Quoted 7 lines
Move the VMX interrupt dispatch magic into the x86 core code. This isolates KVM from the FRED/IDT decisions and reduces the amount of
EXPORT_SYMBOL_FOR_KVM().Suggested-by Sean Christopherson <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>---
<..>
Hi Peter,
I gave this a spin on the CI that hit the original failures, and those tests pass, + no new issues.
Thomas Gleixner
maintainerQuoted 19 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Sean Christopherson
This as delta? (I had typed this all up before Peter posted a new verison, so dammit I'm sending it!)
Quoted 8 lines
Quoted 3 lines
:-)
I'll go stare at it in the morning, I'm about to go crash out.
New delta against your effective v2, builds all of my configs (which isn't _that_ many, but I think they cover most of the weirder ways to include KVM (or not)).
I'll start testing the full thing to try and get early signal on the health.
Oh, re-reading commit
28d11e4548b7, I think this wrecks NMIs. The patch will also use the FRED NMI path and that's not good when running IDT.
Quoted 8 lines
Quoted 1 line
I'll go fix that and stick in a few more comments to clarify this magic.
This should probably we at least two patches, one moving the code into the x86 core and one adding that hrtimer fix on top. But kept as one for now.
Irrespective of the hrtimer fix, I think the initial move into x86 core part is a sane move.
I agree.
Quoted 21 lines
arch/x86/entry/common.c new file +21
-0+#if IS_ENABLED (CONFIG_KVM_INTEL ) +/* +* On VMX, NMIs and IRQs (as configured by KVM) are acknowledge by hardware as +* part of the VM-Exit, i.e. the event itself is consumed as part the VM-Exit. +* x86_entry_from_kvm() is invoked by KVM to effectively forward NMIs and IRQs +* to the kernel for servicing. On SVM, a.k.a. AMD, the NMI/IRQ VM-Exit is +* purely a signal that an NMI/IRQ is pending, i.e. the event that triggered +* the VM-Exit is held pending until it's unblocked in the host. +*/ +noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) +{ +if (event_type ==EVENT_TYPE_EXTINT ) { +#ifdef CONFIG_X86_64 +/* +* Use FRED dispatch, even when running IDT. The dispatch +* tables are kept in sync between FRED and IDT, and the FRED +* dispatch works well with CFI. +*/ +fred_entry_from_kvm (event_type, vector); +#else +idt_entry_from_kvm (vector);
arch/x86/entry/common.c line 39
Quoted 10 lines
arch/x86/entry/common.c new file +10
-0+#endif +/* +* Strictly speaking, only the NMI path requires noinstr. +*/ +instrumentation_begin (); +/* +* KVM/VMX will dispatch from IRQ-disabled but for a context +* that will have IRQs-enabled. This confuses the entry code +* and it will not have reprogrammed the timer (or do +* preemption). Minimal fixup for now.
It's sadly the final fixup.
There is no way that KVM can benefit from the deferred rearm mechanism in case that NEED_RESCHED is set.
That's actually independent of the VMX specific handle_exit_irqoff() situation, which obviously cannot reschedule.
The other early local_irq_enable/disable dance, which caused you to buy a new WTF'o'meter, happens with preemption disabled, so the irq exit code will do the rearm right there.
So be it ....
Thanks,
tglx
Peter Zijlstra
maintainerQuoted 68 lines · Thomas Gleixner
Quoted 19 lines · Peter Zijlstra
Quoted 18 lines · Peter Zijlstra
Quoted 5 lines · Sean Christopherson
Quoted 4 lines · Peter Zijlstra
Quoted 2 lines · Sean Christopherson
This as delta? (I had typed this all up before Peter posted a new verison, so dammit I'm sending it!)
Quoted 8 lines
Quoted 3 lines
:-)
I'll go stare at it in the morning, I'm about to go crash out.
New delta against your effective v2, builds all of my configs (which isn't _that_ many, but I think they cover most of the weirder ways to include KVM (or not)).
I'll start testing the full thing to try and get early signal on the health.
Oh, re-reading commit
28d11e4548b7, I think this wrecks NMIs. The patch will also use the FRED NMI path and that's not good when running IDT.Quoted 8 lines
Quoted 1 line
I'll go fix that and stick in a few more comments to clarify this magic.
This should probably we at least two patches, one moving the code into the x86 core and one adding that hrtimer fix on top. But kept as one for now.
Irrespective of the hrtimer fix, I think the initial move into x86 core part is a sane move.
I agree.
Quoted 21 lines
arch/x86/entry/common.c new file +21
-0+#if IS_ENABLED (CONFIG_KVM_INTEL ) +/* +* On VMX, NMIs and IRQs (as configured by KVM) are acknowledge by hardware as +* part of the VM-Exit, i.e. the event itself is consumed as part the VM-Exit. +* x86_entry_from_kvm() is invoked by KVM to effectively forward NMIs and IRQs +* to the kernel for servicing. On SVM, a.k.a. AMD, the NMI/IRQ VM-Exit is +* purely a signal that an NMI/IRQ is pending, i.e. the event that triggered +* the VM-Exit is held pending until it's unblocked in the host. +*/ +noinstr void x86_entry_from_kvm(unsigned int event_type, unsigned int vector) +{ +if (event_type ==EVENT_TYPE_EXTINT ) { +#ifdef CONFIG_X86_64 +/* +* Use FRED dispatch, even when running IDT. The dispatch +* tables are kept in sync between FRED and IDT, and the FRED +* dispatch works well with CFI. +*/ +fred_entry_from_kvm (event_type, vector); +#else +idt_entry_from_kvm (vector);arch/x86/entry/common.c line 39
Quoted 10 lines
arch/x86/entry/common.c new file +10
-0+#endif +/* +* Strictly speaking, only the NMI path requires noinstr. +*/ +instrumentation_begin (); +/* +* KVM/VMX will dispatch from IRQ-disabled but for a context +* that will have IRQs-enabled. This confuses the entry code +* and it will not have reprogrammed the timer (or do +* preemption). Minimal fixup for now. It's sadly the final fixup.
There is no way that KVM can benefit from the deferred rearm mechanism in case that
NEED_RESCHEDis set.
Quoted 8 lines
That's actually independent of the VMX specific
handle_exit_irqoff()situation, which obviously cannot reschedule.The other early local_irq_enable/disable dance, which caused you to buy a new WTF'o'meter, happens with preemption disabled, so the irq exit code will do the rearm right there.
So be it ....
OK, I'll go make this two patches, fix up that comment and post it proper.
Sean Christopherson
maintainerQuoted 10 lines · Peter Zijlstra
Quoted 9 lines · Sean Christopherson
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Sean Christopherson
This as delta? (I had typed this all up before Peter posted a new verison, so dammit I'm sending it!)
:-)
I'll go stare at it in the morning, I'm about to go crash out.
Quoted 7 lines
Quoted 4 lines
New delta against your effective v2, builds all of my configs (which isn't _that_ many, but I think they cover most of the weirder ways to include KVM (or not)).
I'll start testing the full thing to try and get early signal on the health.
Oh, re-reading commit
28d11e4548b7, I think this wrecks NMIs. The patch will also use the FRED NMI path and that's not good when running IDT.
Oof, good catch. Which in turn explains why KVM's VMX_DO_EVENT_IRQOFF still uses CONFIG_X86_64 instead of CONFIG_X86_FRED.
Paolo Bonzini
maintainerIl mar 21 apr 2026, 19:55 Sean Christopherson <[email protected]> ha scritto:
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
FWIW, VMX should work just like SVM if we clear
VM_EXIT_ACK_INTR_ON_EXIT.Hell no.
Quoted 2 lines
I know. What's the point of that
VM_EXIT_ACK_INTR_ON_EXITexercise? Is there any performance benefit or is it just used because it's there?There are performance benefits, and it preserves ordering [...] NMIs are unconditionally "acked" on VM-Exit.
Not that I disagree but...
Quoted 2 lines
Even if performance is "fine", changing decades of fundamental KVM behavior is terrifying.
... it's not decades, ack on VM exit is actually relatively recent (10 years out 20 :)). The reason why it was introduced is another killer for the idea, though. Posted interrupts require it, for some reason only known to Intel.
Thanks,
Paolo
Sean Christopherson
maintainerQuoted 12 lines · Paolo Bonzini
Il mar 21 apr 2026, 19:55 Sean Christopherson <[email protected]> ha scritto:
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
FWIW, VMX should work just like SVM if we clear
VM_EXIT_ACK_INTR_ON_EXIT.Hell no.
Quoted 2 lines
I know. What's the point of that
VM_EXIT_ACK_INTR_ON_EXITexercise? Is there any performance benefit or is it just used because it's there?There are performance benefits, and it preserves ordering [...] NMIs are unconditionally "acked" on VM-Exit.
Quoted 8 lines
Not that I disagree but...
Quoted 2 lines
Even if performance is "fine", changing decades of fundamental KVM behavior is terrifying.
... it's not decades, ack on VM exit is actually relatively recent (10 years out 20 :)). The reason why it was introduced is another killer for the idea, though. Posted interrupts require it,
Oh, I forgot about posted interrupts. So yeah, what Paolo said :-)
Quoted 1 line
for some reason only known to Intel.
My guess is because the ucode that morphs the notification vector into posted interrupt processing is a sub-clause of the ACK flow. And from an architectural perspective, having the CPU ACK/dismiss some IRQs (notification vectors) but not others (everything else) would be kludgy (even more so than special casing the notification vector already is).
Thomas Gleixner
maintainerQuoted 8 lines · Paolo Bonzini
Il mar 21 apr 2026, 19:55 Sean Christopherson <[email protected]> ha scritto:
Quoted 2 lines
Even if performance is "fine", changing decades of fundamental KVM behavior is terrifying.
... it's not decades, ack on VM exit is actually relatively recent (10 years out 20 :)). The reason why it was introduced is another killer for the idea, though. Posted interrupts require it, for some reason only known to Intel.
That's yet another attempt to provide an halfways decent argument, but as anything else related to KVM/VIRT it is neither documented nor proven to be true.
Your claim is just another prove of the KVM handwaving approach as the change log which introduced this (commit a547c6db4d2f) does not mention it at all. In fact posted interrupt support came _three_ years after that and there is exactly _zero_ information about this dependency if I read the relevant change logs correctly.
Especially commit f2485b3e0c6c ("KVM: x86: use guest_exit_irqoff") which fundamentally changed the behavior from regs::flags::IF = 1 to regs::flags::IF = 0 does not mention this at all:
KVM: x86: use guest_exit_irqoff
This gains a few clock cycles per vmexit. On Intel there is no need anymore to enable the interrupts in vmx_handle_external_intr, since we are using the "acknowledge interrupt on exit" feature. AMD needs to do that, and must be careful to avoid the interrupt shadow.
Written and committed by a Paolo Bonzini dude. You might know him perhaps.
Where exactly is the reference to posted interrupts and how did you or someone else establish that this hack is a requirement for posted interrupts in the first place?
Nice try. Try again.
Thanks,
tglx
Paolo Bonzini
maintainerQuoted 9 lines · Thomas Gleixner
Quoted 4 lines · Paolo Bonzini
... it's not decades, ack on VM exit is actually relatively recent (10 years out 20 :)). The reason why it was introduced is another killer for the idea, though. Posted interrupts require it, for some reason only known to Intel.
That's yet another attempt to provide an halfways decent argument, but as anything else related to KVM/VIRT it is neither documented nor proven to be true.
SDM, "27.2.1 Checks on VMX Controls", under "VM-Execution Control Fields":
If the “process posted interrupts” VM-execution control is 1, the following must be true: [...]
— The “acknowledge interrupt on exit” VM-exit control is 1.
I grant you the "not documented" (not cross referenced to the manual in commit 01e439be775) part, but "not proven to be true"? Neither I nor my predecessors are *that* clueless.
And under "posted-interrupt processing" the following two steps are listed:
1. The local APIC is acknowledged; this provides the processor core with an interrupt vector, called here the physical vector.
2. If the physical vector equals the posted-interrupt notification vector, the logical processor continues to the next step. Otherwise, a VM exit occurs as it would normally due to an external interrupt; the vector is saved in the VM-exit interruption-information field.
So the reason is actually documented by the SDM: "ack intr on exit" is needed in order to retrieve the interrupt vector, which is then compared against the posted-interrupt notification vector. As far as the APIC is concerned, the interrupt is gone; there's no way to "send it back" and have it delivered after IF=1.
Quoted 5 lines
Your claim is just another prove of the KVM handwaving approach as the change log which introduced this (commit
a547c6db4d2f) does not mention it at all. In fact posted interrupt support came _three_ years after that and there is exactly _zero_ information about this dependency if I read the relevant change logs correctly.
No, posted interrupt support on the CPU (not in interrupt remapping) arrived in the same series with commit 5a71785dde30 ("KVM: VMX: Use posted interrupt to deliver virtual interrupt", 2013-04-16).
There were two different steps. Posted interrupt support on the CPU was introduced in Ivy Bridge. Posted interrupt support on the IOMMU in Broadwell (and added to KVM with commit efc644048ecd, "KVM: x86:
Update IRTE for posted-interrupts", 2015-10-01). New virt features used to come on the "tock" instead of the "tick").
Quoted 3 lines
Where exactly is the reference to posted interrupts and how did you or someone else establish that this hack is a requirement for posted interrupts in the first place?
Is reading the manual enough?
Paolo
Quoted 6 lines · Paolo Bonzini
...
There were two different steps. Posted interrupt support on the CPU was introduced in Ivy Bridge. Posted interrupt support on the IOMMU in Broadwell (and added to KVM with commitefc644048ecd, "KVM: x86:
Update IRTE for posted-interrupts", 2015-10-01). New virt features used to come on the "tock" instead of the "tick").
Nit: I believe posted interrupts were only introduced in the higher end Ivy Town parts, but I could be wrong. :)
Paolo Bonzini
maintainerQuoted 11 lines · Jim Mattson
Quoted 6 lines · Paolo Bonzini
...
There were two different steps. Posted interrupt support on the CPU was introduced in Ivy Bridge. Posted interrupt support on the IOMMU in Broadwell (and added to KVM with commitefc644048ecd, "KVM: x86:
Update IRTE for posted-interrupts", 2015-10-01). New virt features used to come on the "tock" instead of the "tick").Nit: I believe posted interrupts were only introduced in the higher end Ivy Town parts, but I could be wrong. :)
Yes, they were fused out in Xeon E3 and below. I was only referring to the processor generation ("v2" in the CPUID model id) - never heard about "Town" until now.
Paolo
Thomas Gleixner
maintainerQuoted 13 lines · Paolo Bonzini
Quoted 9 lines · Thomas Gleixner
Quoted 4 lines · Paolo Bonzini
... it's not decades, ack on VM exit is actually relatively recent (10 years out 20 :)). The reason why it was introduced is another killer for the idea, though. Posted interrupts require it, for some reason only known to Intel.
That's yet another attempt to provide an halfways decent argument, but as anything else related to KVM/VIRT it is neither documented nor proven to be true.
SDM, "27.2.1 Checks on VMX Controls", under "VM-Execution Control Fields":
Quoted 7 lines
If the “process posted interrupts” VM-execution control is 1, the following must be true: [...]
— The “acknowledge interrupt on exit” VM-exit control is 1.I grant you the "not documented" (not cross referenced to the manual in commit
01e439be775) part, but "not proven to be true"? Neither I nor my predecessors are *that* clueless.
I did not claim at all that you or your predecessors are clueless, but do I really have to do all the homework which you and them obviously failed to do?
And even now in this discussion you came only forth with a real explanation after I told you that it's nowhere documented in the kernel code or the related changelogs.
Still now you tell me that 01e439be775 is the key to all of this, which is completely irrelevant because the commit which started to violate the core kernel code semantics and expectations is f2485b3e0c6c ("KVM: x86:
use guest_exit_irqoff") which happened _three_ years _after_ 01e439be775 introduced the detection of posted interrupts.
Do you really expect me to connect the useless changelog of f2485b3e0c6c to the posted interrupts restrictions which are not mentioned anywhere?
Quoted 1 line
Is reading the manual enough?
Are you seriously asking me to read the manual to figure out why some completely undocumented code exists? And then deduce why that code violates basic principles of the core code assumptions which have been there way _before_ any of this x86 virtualization gunk even existed on a napkin sketch?
You are putting the cart before the horse.
KVM is violating basic principles without giving any clue to those who try to keep them maintained. Neither in changelogs nor in code nor in comments.
Please don't misunderstand me. I'm not trying to fingerpoint here. I'm trying to understand how we got into this mess and what's the proper way out of it.
At least after tons of useless wasted cycles/emails I know now that there seems to be some real existing hardware constraint.
As a consequence I'll go and use the irq_regs workaround in hrtimer_interrupt_rearm() for now. It's not pretty, but as I pointed out before, it _is_ covering _all_ (even not yet discovered) KVM/VIRT induced violations of core code semantics and expectations in one go and it is pretty much performance neutral for the price that the KVM/VMX host side can't take advantage of it.
If KVM/VMX want's to benefit from the semantically correct hrtimer/HRTICK related core code improvements, then please post patches which might hit the 7.2 merge window eventually in case they make sense and most importantly are semantically correct.
Thanks,
tglx
Thomas Gleixner
maintainerQuoted 6 lines · Sean Christopherson
Quoted 3 lines · Thomas Gleixner
Quoted 3 lines
Quoted 1 line
Looks like. It will take the interrupt after
local_irq_enable().FWIW, VMX should work just like SVM if we clear
VM_EXIT_ACK_INTR_ON_EXIT.Hell no.
I knew for sure that someone from the KVM camp would cry murder :)
Quoted 8 lines
Quoted 2 lines
I know. What's the point of that
VM_EXIT_ACK_INTR_ON_EXITexercise? Is there any performance benefit or is it just used because it's there?There are performance benefits, and it preserves ordering: the first IRQ that's serviced by the host is guaranteed to be _the_ IRQ that triggered the VM-Exit. E.g. with AMD's approach, any IRQs that arrive between the VM-Exit and STI (which is a pretty big swath of code) could be serviced before the IRQ that triggered the exit, depending on priority.
I might eventually buy the performance benefit, but the ordering is not interesting at all. That's a pure virt-cult fallacy to believe that it matters. Why?
Look at this bare metal scenario with two interrupts A and B where B has a higher priority than A:
cli
interrupt A is raised in the APIC
tons of code
interrupt B is raised in the APIC
sti
handle(B)
handle(A)
or
cli
interrupt A is raised in the APIC
tons of code
sti
handle(A)
interrupt B is raised in the APIC
handle(B)
It's completely uninteresting which one is handled first. Otherwise this 'handle it directly' approach in VMX would not be correct at all.
The only valid argument here is performance and I'm not really convinced that it actually matters given the amount of other nonsense which has to be done on a VMEXIT nowadays.
The point is that the early handling only affects the actual response time to the interrupt itself, but it does not affect the response time to anything the interrupt might trigger which requires interrupt and/or preemption enabled context:
VMENTER
-> Host interrupt
VMEXIT
handle_early()
irqentry_enter()
irq_enter();
handle();
irq_exit(); // Cannot handle soft interrupts because IF = 0
irqentry_exit(); // Cannot handle preemption because IF = 0I understand that this is optimizing for the case where neither soft interrupts nor preemption has to be handled, but all I have seen so far is handwaving about the actual performance benefits. See below.
Quoted 2 lines
VM_EXIT_ACK_INTR_ON_EXITalso provides symmetry with Intel's handing of NMIs, as NMIs are unconditionally "acked" on VM-Exit.
What's the exact point you are trying to make?
The symmetry is a cosmetic nice to have bullet point, but neither a functional nor a correctness requirement. The fact that hardware people provided something which looks "useful" at the first glance does not make it so.
Quoted 2 lines
Even if performance is "fine", changing decades of fundamental KVM behavior is terrifying.
It worked perfectly fine before this was introduced in commit a547c6db4d2f ("KVM: VMX: Enable acknowledge interupt on vmexit") in 2013.
If you decrypt that commit message and read the patch then you'll notice that back then this issue would not have happened at all because the register frame had IF set.
This got changed by f2485b3e0c6c ("KVM: x86: use guest_exit_irqoff") in June 2016 to save an completely unspecified amount of 'few cycles'.
So much for decades and for useful changelogs which actually prove that something has a substantial benefit.
Given the amount of changes since then it would be really interesting to see actual numbers for the benefit of VM_EXIT_ACK_INTR_ON_EXIT before we end up with more KVM/VIRT specific oddities all over the place.
I'm more than mildly amused that you are terrified by the thought of reverting back to something which is known _and_ guaranteed to work while at the same time you are willing to accept any shortcut in the so fundamental KVM behavior to gain a cycle for the price that everything else has to adjust to the semantically broken view of KVM.
There is plenty of proof in the git history that KVM follows the performance first, correctness later principle and I personally have wasted a lot of _my_ precious time due to that since the day KVM was shoved into the kernel, which was actually almost _two_ decades ago.
Quoted 8 lines
Pulling in an earlier idea:
: Now for VMX, that
hrtimer_rearm_deferred()call should really go into :handle_external_interrupt_irqoff(), which in turn requires to export :__hrtimer_rearm_deferred().IMO, that's the way to go. But instead of exporting
__hrtimer_rearm_deferred(), movevmx_do_nmi_irqoff()andvmx_do_interrupt_irqoff()into core kernel entry code
Surely not into core kernel entry code as this is x86 specific hackery.
Quoted 8 lines
(along with the assembly glue), and then
EXPORT_SYMBOL_FOR_KVMthose. It'd mean some extra surgery, e.g. to provide an equivalent to KVM's IDT lookup:
gate_offset((gate_desc *)host_idt_base + vector)But I suspect it would be a big net positive in the end.i E.g. the entry code would *know* it's dealing with a direct call from KVM, and thus shouldn't need to play
pt_regsgames.
As this is x86 specific the generic entry code knows absolutely nothing unless there is a magic indicator like PeterZ's hack or yet another duplicated version of the irqentry_exit() code just to accomodate KVM for handwaving reasons.
As Peter and myself pointed out before this will also not solve the problem that due to that KVM won't be able to benefit from the recent hrtimer/hrtick improvements on VMX(TDX) hosts.
To be entirely clear: We are not going to disable HRTICK for the benefit of this dubious "decades old performance" hack.
Quoted 4 lines
Actually, even better would be to bury the FRED vs. not-FRED details in entry code. E.g. on the KVM invocation side, we could get to something like the below, and I'm pretty sure _reduce_ the number of for-KVM exports in the process.
That's an orthogonal issue. The problem at hand is independent of FRED or not-FRED as both end up providing a pt_regs frame with eflags.IF = 0.
For the short term fix, which is required no matter what, checking irq_regs in hrtimer_interrupt_rearm() is not the worst solution as it covers _all_ not yet unearthed issues which are nicely hidden in some dusty corners of architecture specific KVM optimizations and will only come out around 7.1-rc7 or later when people actually can be bothered to test stuff...
I just booted a big machine with that patch applied. get_irq_regs() and the regs_irqs_disabled() check are barely visible in perf because the cache line is 99% of the time hot and as it is strictly per CPU there is no contention at all. The only case where it shows up is when there is a massive amount of hrtimers to expire at the same time with D-cache consuming callbacks. But in that case the extra cache miss of get_irq_regs() is just in the noise and not really relevant.
So far that deferred reprogram mechanism seems to be the only known mechanism which relies on the irqentry_exit() pt_regs::flags::IF state being correct, but in the long run that's not a sustainable solution.
You really want to come up with real numbers which prove the performance benefit to justify the extra complexity of this.
Thanks,
tglx
Sean Christopherson
maintainerQuoted 7 lines · Thomas Gleixner
Quoted 4 lines · Sean Christopherson
Quoted 1 line · Thomas Gleixner
Quoted 1 line
Quoted 1 line
Looks like. It will take the interrupt after
local_irq_enable().
VM_EXIT_ACK_INTR_ON_EXITalso provides symmetry with Intel's handing of NMIs, as NMIs are unconditionally "acked" on VM-Exit.What's the exact point you are trying to make?
That no matter what we do for IRQs, KVM needs a direct call into the kernel to handle an asynchronous event that arrived in the past.
Quoted 10 lines
The symmetry is a cosmetic nice to have bullet point, but neither a functional nor a correctness requirement. The fact that hardware people provided something which looks "useful" at the first glance does not make it so.
Quoted 2 lines
Even if performance is "fine", changing decades of fundamental KVM behavior is terrifying.
It worked perfectly fine before this was introduced in commit
a547c6db4d2f("KVM: VMX: Enable acknowledge interupt on vmexit") in 2013.
Yes, but that configuration hasn't been tested (by KVM) on any CPU released in the last decade+. That's what scares me. Do I think it's at all likely that there's a lurking ucode bug? No. But the risk vs. reward isn't there for me.
But as Paolo pointed out, the "killer" feature gated by ACK-on-exit is posted interrupts, and _that_ provides a massive performance win.
Quoted 4 lines
Quoted 2 lines
IMO, that's the way to go. But instead of exporting
__hrtimer_rearm_deferred(), movevmx_do_nmi_irqoff()andvmx_do_interrupt_irqoff()into core kernel entry codeSurely not into core kernel entry code as this is x86 specific hackery.
Oh come on. I have a hard time believing that you really truly thought that's what I was suggesting.
Quoted 9 lines
Quoted 8 lines
(along with the assembly glue), and then
EXPORT_SYMBOL_FOR_KVMthose. It'd mean some extra surgery, e.g. to provide an equivalent to KVM's IDT lookup:
gate_offset((gate_desc *)host_idt_base + vector)But I suspect it would be a big net positive in the end.i E.g. the entry code would *know* it's dealing with a direct call from KVM, and thus shouldn't need to play
pt_regsgames.
Quoted 8 lines
As this is x86 specific the generic entry code knows absolutely nothing unless there is a magic indicator like PeterZ's hack or yet another duplicated version of the
irqentry_exit()code just to accomodate KVM for handwaving reasons.As Peter and myself pointed out before this will also not solve the problem that due to that KVM won't be able to benefit from the recent hrtimer/hrtick improvements on VMX(TDX) hosts.
Sorry, you lost me here. What's the TDX angle? Or are you just saying that VMX is currently hosed with the deferred rearm?
Quoted 2 lines
To be entirely clear: We are not going to disable HRTICK for the benefit of this dubious "decades old performance" hack.
No one suggested that.
Quoted 7 lines
Quoted 4 lines
Actually, even better would be to bury the FRED vs. not-FRED details in entry code. E.g. on the KVM invocation side, we could get to something like the below, and I'm pretty sure _reduce_ the number of for-KVM exports in the process.
That's an orthogonal issue. The problem at hand is independent of FRED or not-FRED as both end up providing a
pt_regsframe with eflags.IF = 0.
Eh, not if it gives us a clean, maintable solution for for the problem.
Paolo Bonzini
maintainerQuoted 2 lines · Thomas Gleixner
This got changed by
f2485b3e0c6c("KVM: x86: useguest_exit_irqoff") in June 2016 to save an completely unspecified amount of 'few cycles'.
Sean answered in way too much detail :) but since I was about to do it too that is a local_irq_save/restore pair, so 20-50 cycles per vmexit. Nowadays, due to the overhead from mitigating side channel attacks, the cost of vmexits went up from ~1000 clock cycles to (depending on your settings) even 2500-3000.
Back when posted interrupt support was added, "ack intr on exit" was always used when available, but was optional. In 2016 I made it mandatory. The micro-optimization at the time had a consistent 2-3% gain on some benchmarks like UDP_RR, but also on realistic workloads with lots of thread wakeups; the simplification was also a nice bonus, but these days this part is almost negligible these days.
However the optimization that Sean remembered is only a small part of the story. The real issue is that "ack intr on exit" is a requirement for posted intr support. For each interrupt from assigned devices, it saves a full vmexit/vmentry for the vCPU. Adding the extra latency from processing the interrupt, writing to the eventfd and triggering the vmexit, that's at least a couple microseconds in total. That's two orders of magnitude more than the "few cycles" that commit f2485b3e0c6c added on top.
The reference to TDX is not clear to me either, but I am probably missing something obvious because it's a bit late here.
Paolo
Quoted 2 lines
You really want to come up with real numbers which prove the performance benefit to justify the extra complexity of this.
Quoted 6 lines · Thomas Gleixner
Now for VMX, that
hrtimer_rearm_deferred()call should really go intohandle_external_interrupt_irqoff(), which in turn requires to export__hrtimer_rearm_deferred().But we can avoid that alltogether. Something like the untested below.
Tested with the below patch and the tests pass with this too.
Quoted 42 lines
Thanks,
tglx
---kernel/time/hrtimer.c +0
-1@@ -42,9 +42,10 @@ #include <linux/timer.h> #include <linux/freezer.h> #include <linux/compat.h> - #include <linux/uaccess.h>kernel/time/hrtimer.c +2
-0+#include <asm/irq_regs.h> +#include <
trace/events/timer.h>
#include "tick-internal.h"kernel/time/hrtimer.c +0
-0@@ -2062,11 +2063,16 @@ void __hrtimer_rearm_deferred(void)static
__always_inlinevoid
hrtimer_interrupt_rearm(structhrtimer_cpu_base*cpu_base,ktime_texpires_next) {kernel/time/hrtimer.c +10
-5-/* hrtimer_interrupt() just re-evaluated the first expiring timer */ - cpu_base->deferred_needs_update = false; -/* Cache the expiry time */ - cpu_base->deferred_expires_next = expires_next; -set_thread_flag (TIF_HRTIMER_REARM ); +/* Lies, damned lies and virt */ +if (likely (!regs_irqs_disabled (get_irq_regs ()))) { +/* hrtimer_interrupt() just re-evaluated the first expiring timer */ + cpu_base->deferred_needs_update = false; +/* Cache the expiry time */ + cpu_base->deferred_expires_next = expires_next; +set_thread_flag (TIF_HRTIMER_REARM ); + }else { +hrtimer_rearm (cpu_base, expires_next, false); + }}
#else /*CONFIG_HRTIMER_REARM_DEFERRED*/
static__always_inlinevoid
Thomas Gleixner
maintainerQuoted 16 lines · Peter Zijlstra
Quoted 4 lines · Peter Zijlstra
Quoted 4 lines
include/linux/irq-entry-common.h +3
-0/* +* This is sadly required due to KVM, which invokes regular +* interrupt handlers with interrupt disabled state in @regs. +*/ Quoted 8 lines
include/linux/irq-entry-common.h irqentry_exit_to_kernel_mode_after_preem · line 529
Quoted 8 lines
include/linux/irq-entry-common.h +5
-0+instrumentation_begin (); +hrtimer_rearm_deferred (); +instrumentation_end (); + +/* * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry. */ Ohhh, wait. What happens if you take a page-fault from NMI context? Does this then not result in trying to program the timer from NMI context?
Uuuurgh, yes.
Verma, Vishal L
Tested-byQuoted 28 lines · Thomas Gleixner
Subject: entry: Enforce hrtimer rearming in the
irqentry_exitpath
From: Thomas Gleixner <[email protected]>
Date: Tue, 21 Apr 2026 09:00:52 +0200
irqentry_exit_to_kernel_mode_after_preempt()invokeshrtimer_rearm_deferred()only when the interrupted context had interrupts enabled. That's a correct decision because the timer interrupt can only be delivered in interrupt enabled contexts. The interrupt disabled path is used by exceptions and traps which never touch the hrtimer mechanics.So much for the theory, but then there is VIRT which ruins everything.
KVM invokes regular interrupts with
pt_regswhich have interrupts disabled. That's correct from the KVM point of view, but completely violates the obviously correct expectations of the interrupt entry/exit code.Cure this by adding a
hrtimer_rearm_deferred()invocation into the interrupted context has interrupt disabled path ofirqentry_exit_to_kernel_mode_after_preempt().That's unfortunate when there is an actual reschedule pending, but it can't be avoided because KVM invokes a lot of code and also reenables interrupts _before_ reaching the point where the reschedule condition is handled. That can delay the rearming significantly, which in turn can cause artificial latencies.
Quoted 4 lines
Fixes 0e98eb14814e ("entry: Prepare for deferred hrtimer rearming")Reported-by "Verma, Vishal L" <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>
Hi Thomas, I tested this and verified it solves both the tests, no more lockups. If this is the final fix, you can add:
(I'm queueing up Peter's patch on the CI now too)
Quoted 21 lines
---
include/linux/irq-entry-common.h | 8 ++++++++1 file changed, 8 insertions(+)include/linux/irq-entry-common.h +8
-0@@ -516,6 +516,14 @@ irqentry_exit_to_kernel_mode_after_preem instrumentation_end(); } else { /* +* This is sadly required due to KVM, which invokes regular +* interrupt handlers with interrupt disabled state in @regs. +*/ +instrumentation_begin (); +hrtimer_rearm_deferred (); +instrumentation_end (); + +/* * IRQ flags state is correct already. Just tell RCU if it * was not watching on entry. */