x86/entry: Rework leftovers (was part V)
Subsystems (11)
v9 cover x86/entry: Rework leftovers (was part V)
Folks!
This is V9 of the rework series. V7 and V8 were never posted but I used the version numbers for tags while fixing up 0day complaints. The last posted version was V6 which can be found here:
https://lore.kernel.org/r/[email protected]
The V9 leftover series is based on:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry
That branch contains the merged part 1-4 of the original 5 part series.
V9 has the following changes vs. V6:
- Rebase on tip x86/entry
- Simplified the hardware latency detector changes by moving the invocation to the right place in nmi_enter/exit() and annotate it.
- Reworked the conditional RCU handling so it is now used unconditionally everywhere. That simplified the idtentry_enter/exit code significantly and also allowed to simplify the XEN hypercall voluntary preemption handling.
- Moved the run on irq stack logic into an inline to avoid having the same conditionals all over the place and fixed up the relevant places.
- Picked up Acked-by and Reviewed-by tags where appropriate.
The full series is available from:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-v9-the-rest
If we agree on the RCU changes, then these will be applied into the core/rcu branch first so Paul can pick them up to avoid the next conflict horrors.
Thanks,
tglx
PATCH v9 1/39 nmi, tracing: Make hardware latency tracing noinstr safe
The hardware latency tracer calls into instrumentable functions. Move the calls into the RCU watching sections and annotate them.
---
V9: Simplified version
--- include/linux/hardirq.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
include/linux/hardirq.h +6 -2
@@ -82,20 +82,24 @@ extern void irq_exit(void);arch_nmi_enter (); \printk_nmi_enter (); \lockdep_off (); \ -ftrace_nmi_enter (); \BUG_ON (in_nmi () ==NMI_MASK ); \__preempt_count_add (NMI_OFFSET +HARDIRQ_OFFSET ); \rcu_nmi_enter (); \lockdep_hardirq_enter (); \ +instrumentation_begin (); \ +ftrace_nmi_enter (); \ +instrumentation_end (); \ } while (0)#define nmi_exit () \ do { \ + instrumentation_begin(); \ + ftrace_nmi_exit(); \ + instrumentation_end(); \ lockdep_hardirq_exit(); \ rcu_nmi_exit(); \ BUG_ON(!in_nmi()); \ __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ - ftrace_nmi_exit(); \ lockdep_on(); \ printk_nmi_exit(); \ arch_nmi_exit(); \
[tip: x86/entry] nmi, tracing: Make hardware latency tracing noinstr safe
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 1e826b1d2265d1e8ba5f51827a8dba2aa0295f6c Gitweb: https://git.kernel.org/tip/1e826b1d2265d1e8ba5f51827a8dba2aa0295f6c Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:14 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
nmi, tracing: Make hardware latency tracing noinstr safe
The hardware latency tracer calls into instrumentable functions. Move the calls into the RCU watching sections and annotate them.
--- include/linux/hardirq.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
include/linux/hardirq.h +6 -2
@@ -87,20 +87,24 @@ extern void rcu_nmi_exit(void);arch_nmi_enter (); \printk_nmi_enter (); \lockdep_off (); \ -ftrace_nmi_enter (); \BUG_ON (in_nmi () ==NMI_MASK ); \__preempt_count_add (NMI_OFFSET +HARDIRQ_OFFSET ); \rcu_nmi_enter (); \lockdep_hardirq_enter (); \ +instrumentation_begin (); \ +ftrace_nmi_enter (); \ +instrumentation_end (); \ } while (0)#define nmi_exit () \ do { \ + instrumentation_begin(); \ + ftrace_nmi_exit(); \ + instrumentation_end(); \ lockdep_hardirq_exit(); \ rcu_nmi_exit(); \ BUG_ON(!in_nmi()); \ __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ - ftrace_nmi_exit(); \ lockdep_on(); \ printk_nmi_exit(); \ arch_nmi_exit(); \
PATCH v9 2/39 rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()
In reply to Thomas GleixnerFrom: Paul E. McKenney <[email protected]>
There will likely be exception handlers that can sleep, which rules out the usual approach of invoking rcu_nmi_enter() on entry and also rcu_nmi_exit() on all exit paths. However, the alternative approach of just not calling anything can prevent RCU from coaxing quiescent states from nohz_full CPUs that are looping in the kernel: RCU must instead IPI them explicitly. It would be better to enable the scheduler tick on such CPUs to interact with RCU in a lighter-weight manner, and this enabling is one of the things that rcu_nmi_enter() currently does.
What is needed is something that helps RCU coax quiescent states while not preventing subsequent sleeps. This commit therefore splits out the nohz_full scheduler-tick enabling from the rest of the rcu_nmi_enter() logic into a new function named rcu_irq_enter_check_tick().
[ tglx: Renamed the function and made it a nop when context tracking is off ]
---
V9: New patch
--- include/linux/hardirq.h | 9 +++++ kernel/rcu/tree.c | 82 ++++++++++++++++++++++++++++++++++++------------ 2 files changed, 71 insertions(+), 20 deletions(-)
include/linux/hardirq.h +9 -0
@@ -2,6 +2,7 @@#ifndef LINUX_HARDIRQ_H #define LINUX_HARDIRQ_H +#include <linux/context_tracking_state.h> #include <linux/preempt.h> #include <linux/lockdep.h> #include <linux/ftrace_irq.h>
@@ -27,6 +28,14 @@ extern void rcu_nmi_enter(void);extern void rcu_nmi_exit (void );#endif +void __rcu_irq_enter_check_tick (void ); + +static __always_inline void rcu_irq_enter_check_tick(void) +{ +if (context_tracking_enabled ()) +__rcu_irq_enter_check_tick (); +} +/* * It is safe to do non-atomic ops on ->hardirq_context, * because NMI handlers may not preempt and the ops are
kernel/rcu/tree.c +62 -20
94 lines not on this page · raw message
Paul E. McKenney
maintainerRe: [patch V9 02/39] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()
Quoted 11 lines · Thomas Gleixner
From: Paul E. McKenney <[email protected]>
There will likely be exception handlers that can sleep, which rules out the usual approach of invoking
rcu_nmi_enter()on entry and alsorcu_nmi_exit()on all exit paths. However, the alternative approach of just not calling anything can prevent RCU from coaxing quiescent states fromnohz_fullCPUs that are looping in the kernel: RCU must instead IPI them explicitly. It would be better to enable the scheduler tick on such CPUs to interact with RCU in a lighter-weight manner, and this enabling is one of the things thatrcu_nmi_enter()currently does.
Quoted 6 lines
What is needed is something that helps RCU coax quiescent states while not preventing subsequent sleeps. This commit therefore splits out the
nohz_fullscheduler-tick enabling from the rest of thercu_nmi_enter()logic into a new function namedrcu_irq_enter_check_tick().[ tglx: Renamed the function and made it a nop when context tracking is off ]
The new name works for me! A couple of nits called out below.
Thanx, Paul
Quoted 21 lines
Suggested-by Andy Lutomirski <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>---
V9: New patch--- include/linux/hardirq.h | 9 +++++ kernel/rcu/tree.c | 82 ++++++++++++++++++++++++++++++++++++------------ 2 files changed, 71 insertions(+), 20 deletions(-)include/linux/hardirq.h +1
-0@@ -2,6 +2,7 @@#ifndef LINUX_HARDIRQ_H #define LINUX_HARDIRQ_H +#include <linux/context_tracking_state.h> #include <linux/preempt.h> #include <linux/lockdep.h> #include <linux/ftrace_irq.h>
include/linux/hardirq.h extern void rcu_nmi_enter(void); · line 36
Quoted 9 lines
include/linux/hardirq.h +6
-0@@ -27,6 +28,14 @@ extern void rcu_nmi_enter(void);extern void rcu_nmi_exit (void );#endif +void __rcu_irq_enter_check_tick (void ); + +static __always_inline void rcu_irq_enter_check_tick(void) +{ +if (context_tracking_enabled ()) +__rcu_irq_enter_check_tick ();
I suggest moving the WARN_ON_ONCE(in_nmi()) check here to avoid calling in_nmi() twice. Because of the READ_ONCE(), the compiler cannot (had better not!) eliminate the double call.
Quoted 33 lines
include/linux/hardirq.h +2
-0+} +/* * It is safe to do non-atomic ops on ->hardirq_context, * because NMI handlers may not preempt and the ops are kernel/rcu/tree.c +22
-0@@ -848,6 +848,67 @@ void noinstr rcu_user_exit(void) {rcu_eqs_exit (1 ); } + +/** +* __rcu_irq_enter_check_tick - Enable scheduler tick on CPU if RCU needs it. +* +* The scheduler tick is not normally enabled when CPUs enter the kernel +* from nohz_full userspace execution. After all, nohz_full userspace +* execution is an RCU quiescent state and the time executing in the kernel +* is quite short. Except of course when it isn't. And it is not hard to +* cause a large system to spend tens of seconds or even minutes looping +* in the kernel, which can cause a number of problems, include RCU CPU +* stall warnings. +* +* Therefore, if a nohz_full CPU fails to report a quiescent state +* in a timely manner, the RCU grace-period kthread sets that CPU's +* ->rcu_urgent_qs flag with the expectation that the next interrupt or +* exception will invoke this function, which will turn on the scheduler +* tick, which will enable RCU to detect that CPU's quiescent states, +* for example, due to cond_resched() calls in CONFIG_PREEMPT=n kernels. +* The tick will be disabled once a quiescent state is reported for +* this CPU. +* +* Of course, in carefully tuned systems, there might never be an
kernel/rcu/tree.c void noinstr rcu_user_exit(void) · line 882
Quoted 10 lines
kernel/rcu/tree.c +10
-0+* interrupt or exception. In that case, the RCU grace-period kthread +* will eventually cause one to happen. However, in less carefully +* controlled environments, this function allows RCU to get what it +* needs without creating otherwise useless interruptions. +*/ +void __rcu_irq_enter_check_tick (void ) +{ +struct rcu_data *rdp =this_cpu_ptr (&rcu_data); + +// Enabling the tick is unsafe in NMI handlers.
There is an extra space before the "//", probably the one that used to be after the ";" below. ;-)
kernel/rcu/tree.c void noinstr rcu_user_exit(void) · line 887
Quoted 5 lines
kernel/rcu/tree.c +5
-0+if (WARN_ON_ONCE (in_nmi ())) +return ; + +RCU_LOCKDEP_WARN (rcu_dynticks_curr_cpu_in_eqs (), +"Illegal rcu_irq_enter_check_tick() from extended quiescent state" );
The instrumentation_begin() has disappeared, presumably because instrumentation is already enabled in the non-RCU code that directly calls rcu_irq_enter_check_tick(). (I do see the calls in rcu_nmi_enter() below.)
Quoted 28 lines
kernel/rcu/tree.c +24
-0+ +if (!tick_nohz_full_cpu (rdp->cpu) || + !READ_ONCE (rdp->rcu_urgent_qs) || +READ_ONCE (rdp->rcu_forced_tick)) { +// RCU doesn't need nohz_full help from this CPU, or it is +// already getting that help. +return ; + } + +// We get here only when not in an extended quiescent state and +// from interrupts (as opposed to NMIs). Therefore, (1) RCU is +// already watching and (2) The fact that we are in an interrupt +// handler and that the rcu_node lock is an irq-disabled lock +// prevents self-deadlock. So we can safely recheck under the lock. +// Note that the nohz_full state currently cannot change. +raw_spin_lock_rcu_node (rdp->mynode); +if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) { +// A nohz_full CPU is in the kernel and RCU needs a +// quiescent state. Turn on the tick! +WRITE_ONCE (rdp->rcu_forced_tick, true); +tick_dep_set_cpu (rdp->cpu,TICK_DEP_BIT_RCU ); + } + raw_spin_unlock_rcu_node(rdp->mynode); +}#endif /* CONFIG_NO_HZ_FULL */ /**
kernel/rcu/tree.c noinstr void rcu_nmi_enter(void) · line 956
Quoted 2 lines
kernel/rcu/tree.c +0
-0@@ -894,26 +955,7 @@ noinstr void rcu_nmi_enter(void) incby =1 ; }else if (!in_nmi ()) {
This can just be "else" given the in_nmi() check in __rcu_irq_enter_check_tick(), right? Ah, that check got a WARN_ON_ONCE(), so never mind!
I guess that will discourage NMI-handler calls torcu_irq_enter_check_tick(). ;-)
It does mean a double call to in_nmi(), though, so should that WARN_ON_ONCE(in_nmi()) check go into the rcu_irq_enter_check_tick() wrapper? Or do modern compilers figure this one out? Given the READ_ONCE() in preempt_count(), I have to say that I hope not. So see my comment above on rcu_irq_enter_check_tick().
Quoted 26 lines
kernel/rcu/tree.c +1
-20instrumentation_begin (); -if (tick_nohz_full_cpu (rdp->cpu) && - rdp->dynticks_nmi_nesting ==DYNTICK_IRQ_NONIDLE && -READ_ONCE (rdp->rcu_urgent_qs) && - !READ_ONCE (rdp->rcu_forced_tick)) { -// We get here only if we had already exited the -// extended quiescent state and this was an -// interrupt (not an NMI). Therefore, (1) RCU is -// already watching and (2) The fact that we are in -// an interrupt handler and that the rcu_node lock -// is an irq-disabled lock prevents self-deadlock. -// So we can safely recheck under the lock. -raw_spin_lock_rcu_node (rdp->mynode); -if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) { -// A nohz_full CPU is in the kernel and RCU -// needs a quiescent state. Turn on the tick! -WRITE_ONCE (rdp->rcu_forced_tick, true); -tick_dep_set_cpu (rdp->cpu,TICK_DEP_BIT_RCU ); - } -raw_spin_unlock_rcu_node (rdp->mynode); - } +rcu_irq_enter_check_tick ();instrumentation_end (); }instrumentation_begin ();
Re: [patch V9 02/39] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()
Quoted 11 lines · Paul E. McKenney
include/linux/hardirq.h extern void rcu_nmi_enter(void); · line 36
Quoted 6 lines · Thomas Gleixner
include/linux/hardirq.h +6
-0+void __rcu_irq_enter_check_tick (void ); + +static __always_inline void rcu_irq_enter_check_tick(void) +{ +if (context_tracking_enabled ()) +__rcu_irq_enter_check_tick ();I suggest moving the WARN_ON_ONCE(in_nmi()) check here to avoid calling
in_nmi()twice. Because of theREAD_ONCE(), the compiler cannot (had better not!) eliminate the double call.
Makes sense.
Quoted 8 lines
kernel/rcu/tree.c void noinstr rcu_user_exit(void) · line 882
Quoted 5 lines
kernel/rcu/tree.c +5
-0+void __rcu_irq_enter_check_tick (void ) +{ +struct rcu_data *rdp =this_cpu_ptr (&rcu_data); + +// Enabling the tick is unsafe in NMI handlers. There is an extra space before the "//", probably the one that used to be after the ";" below. ;-)
This is caused by my fundamental and not suppressible disgust of tail comments. They really disturb the reading flow for me.
if (foo)
return; // Because ...makes my pattern recognition stop because the semicolon is usually the end of the statement. But that's not the only reason.
// Because ....
if (foo)
return;makes more sense to me because then the comment is explaining the condition and not the outcome. The outcome is obvious when the condition is well explained.
There are a few exceptions where I adjusted, e.g. in macros:
foo(); \
bar_or_something_else(); \but only when the trailing backslash is properly aligned.
foo(); \
bar_or_something_else(); \That stops the parser as well.
I know that this is a pet pieve but I can't help it to adjust it when I have a chance to do so :)
Quoted 10 lines
kernel/rcu/tree.c void noinstr rcu_user_exit(void) · line 887
Quoted 5 lines
kernel/rcu/tree.c +5
-0+if (WARN_ON_ONCE (in_nmi ())) +return ; + +RCU_LOCKDEP_WARN (rcu_dynticks_curr_cpu_in_eqs (), +"Illegal rcu_irq_enter_check_tick() from extended quiescent state" );The
instrumentation_begin()has disappeared, presumably because instrumentation is already enabled in the non-RCU code that directly callsrcu_irq_enter_check_tick(). (I do see the calls inrcu_nmi_enter()below.)
Yes. The intention here is to make sure that the caller does not misplace it. So if the call is in a non-instrumentable code path then objtool will complain and the developer will hopefully think twice whether this is the right place to wrap the call with instrumentation_* annotations. I know it's based on hope :)
Quoted 35 lines
Quoted 28 lines
kernel/rcu/tree.c +24
-0+ +if (!tick_nohz_full_cpu (rdp->cpu) || + !READ_ONCE (rdp->rcu_urgent_qs) || +READ_ONCE (rdp->rcu_forced_tick)) { +// RCU doesn't need nohz_full help from this CPU, or it is +// already getting that help. +return ; + } + +// We get here only when not in an extended quiescent state and +// from interrupts (as opposed to NMIs). Therefore, (1) RCU is +// already watching and (2) The fact that we are in an interrupt +// handler and that the rcu_node lock is an irq-disabled lock +// prevents self-deadlock. So we can safely recheck under the lock. +// Note that the nohz_full state currently cannot change. +raw_spin_lock_rcu_node (rdp->mynode); +if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) { +// A nohz_full CPU is in the kernel and RCU needs a +// quiescent state. Turn on the tick! +WRITE_ONCE (rdp->rcu_forced_tick, true); +tick_dep_set_cpu (rdp->cpu,TICK_DEP_BIT_RCU ); + } + raw_spin_unlock_rcu_node(rdp->mynode); +}#endif /* CONFIG_NO_HZ_FULL */ /** kernel/rcu/tree.c noinstr void rcu_nmi_enter(void) · line 956
Quoted 2 lines
kernel/rcu/tree.c +0
-0@@ -894,26 +955,7 @@ noinstr void rcu_nmi_enter(void) incby =1 ; }else if (!in_nmi ()) {This can just be "else" given the
in_nmi()check in__rcu_irq_enter_check_tick(), right? Ah, that check got aWARN_ON_ONCE(), so never mind!
Quoted 2 lines
I guess that will discourage NMI-handler calls to
rcu_irq_enter_check_tick(). ;-)
Exactly.
Quoted 5 lines
It does mean a double call to
in_nmi(), though, so should that WARN_ON_ONCE(in_nmi()) check go into thercu_irq_enter_check_tick()wrapper? Or do modern compilers figure this one out? Given theREAD_ONCE()inpreempt_count(), I have to say that I hope not. So see my comment above onrcu_irq_enter_check_tick().
Moving it to the wrapper is the right thing to do. Will fix.
Thanks,
tglx
Re: [patch V9 02/39] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()
In reply to Paul E. McKenneyQuoted 14 lines · Paul E. McKenney
Quoted 14 lines
(quoted diff) +14
-0+ if (!tick_nohz_full_cpu(rdp->cpu) || + !READ_ONCE(rdp->rcu_urgent_qs) || + READ_ONCE(rdp->rcu_forced_tick)) { + // RCU doesn't need nohz_full help from this CPU, or it is + // already getting that help. + return; + } + + // We get here only when not in an extended quiescent state and + // from interrupts (as opposed to NMIs). Therefore, (1) RCU is + // already watching and (2) The fact that we are in an interrupt + // handler and that the rcu_node lock is an irq-disabled lock + // prevents self-deadlock. So we can safely recheck under the lock. + // Note that the nohz_full state currently cannot change.
Quoted 8 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+ raw_spin_lock_rcu_node(rdp->mynode); + if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) { + // A nohz_full CPU is in the kernel and RCU needs a + // quiescent state. Turn on the tick! + WRITE_ONCE(rdp->rcu_forced_tick, true); + tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU); + } + raw_spin_unlock_rcu_node(rdp->mynode);
BTW., can we please not ever use this weird comment style in the future?
Linus gave an exception to single-line C++ style comments - but I don't think that should be extrapolated to a license to uglify the kernel with inconsistent muck like this. :-/
I've sanitized it via the patch below.
( I also fixed the whitespace damage and a capitalization typo while
at it, and fixed the spelling in the big comment explaining __rcu_irq_enter_check_tick(). )
Thanks,
Ingo
kernel/rcu/tree.c +20 -14
@@ -850,14 +850,14 @@ void noinstr rcu_user_exit(void) }/** -* __rcu_irq_enter_check_tick - Enable scheduler tick on CPU if RCU needs it. +* __rcu_irq_enter_check_tick - Enable the scheduler tick on a CPU if RCU needs it. * * The scheduler tick is not normally enabled when CPUs enter the kernel * from nohz_full userspace execution. After all, nohz_full userspace * execution is an RCU quiescent state and the time executing in the kernel -* is quite short. Except of course when it isn't. And it is not hard to +* is quite short. Except of course when it isn't: it is not hard to * cause a large system to spend tens of seconds or even minutes looping -* in the kernel, which can cause a number of problems, include RCU CPU +* in the kernel, which can cause a number of problems, including RCU CPU * stall warnings. * * Therefore, if a nohz_full CPU fails to report a quiescent state
@@ -879,7 +879,7 @@ void __rcu_irq_enter_check_tick(void) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data); -// Enabling the tick is unsafe in NMI handlers. +/* Enabling the tick is unsafe in NMI handlers. */ if (WARN_ON_ONCE (in_nmi ()))return ;
@@ -889,21 +889,27 @@ void __rcu_irq_enter_check_tick(void)if (!tick_nohz_full_cpu (rdp->cpu) || !READ_ONCE (rdp->rcu_urgent_qs) ||READ_ONCE (rdp->rcu_forced_tick)) { -// RCU doesn't need nohz_full help from this CPU, or it is -// already getting that help. +/* +* RCU doesn't need nohz_full help from this CPU, or it is +* already getting that help. +*/ return ; } -// We get here only when not in an extended quiescent state and -// from interrupts (as opposed to NMIs). Therefore, (1) RCU is -// already watching and (2) The fact that we are in an interrupt -// handler and that the rcu_node lock is an irq-disabled lock -// prevents self-deadlock. So we can safely recheck under the lock. -// Note that the nohz_full state currently cannot change. +/* +* We get here only when not in an extended quiescent state and +* from interrupts (as opposed to NMIs). Therefore, (1) RCU is +* already watching and (2) the fact that we are in an interrupt +* handler and that the rcu_node lock is an irq-disabled lock +* prevents self-deadlock. So we can safely recheck under the lock. +* Note that the nohz_full state currently cannot change. +*/ raw_spin_lock_rcu_node (rdp->mynode);if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) { -// A nohz_full CPU is in the kernel and RCU needs a -// quiescent state. Turn on the tick! +/* +* A nohz_full CPU is in the kernel and RCU needs a +* quiescent state. Turn on the tick! +*/ WRITE_ONCE (rdp->rcu_forced_tick, true);tick_dep_set_cpu (rdp->cpu,TICK_DEP_BIT_RCU ); }
Paul E. McKenney
maintainerRe: [patch V9 02/39] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()
Quoted 28 lines · Ingo Molnar
Quoted 14 lines · Paul E. McKenney
Quoted 14 lines
(quoted diff) +14
-0+ if (!tick_nohz_full_cpu(rdp->cpu) || + !READ_ONCE(rdp->rcu_urgent_qs) || + READ_ONCE(rdp->rcu_forced_tick)) { + // RCU doesn't need nohz_full help from this CPU, or it is + // already getting that help. + return; + } + + // We get here only when not in an extended quiescent state and + // from interrupts (as opposed to NMIs). Therefore, (1) RCU is + // already watching and (2) The fact that we are in an interrupt + // handler and that the rcu_node lock is an irq-disabled lock + // prevents self-deadlock. So we can safely recheck under the lock. + // Note that the nohz_full state currently cannot change.Quoted 8 lines
Review
Quoted 8 lines
(quoted diff) +8
-0+ raw_spin_lock_rcu_node(rdp->mynode); + if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) { + // A nohz_full CPU is in the kernel and RCU needs a + // quiescent state. Turn on the tick! + WRITE_ONCE(rdp->rcu_forced_tick, true); + tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU); + } + raw_spin_unlock_rcu_node(rdp->mynode);BTW., can we please not ever use this weird comment style in the future?
Quoted 5 lines
Linus gave an exception to single-line C++ style comments - but I don't think that should be extrapolated to a license to uglify the kernel with inconsistent muck like this. :-/
I've sanitized it via the patch below.
The "//" comment style does save vertical space. Is it really ugly or just unfamiliar? For purposes of comparison, back in the day, the "/* */" style seemed quite strange compared to my earlier languages' commenting styles.
Quoted 3 lines
( I also fixed the whitespace damage and a capitalization typo while
at it, and fixed the spelling in the big comment explaining__rcu_irq_enter_check_tick(). )
Some were stylistic rather than wrong, but I have no objection to any of these changes.
Thanx, Paul
Quoted 71 lines
Thanks,
Ingo
kernel/rcu/tree.c +20
-14@@ -850,14 +850,14 @@ void noinstr rcu_user_exit(void) }/** -* __rcu_irq_enter_check_tick - Enable scheduler tick on CPU if RCU needs it. +* __rcu_irq_enter_check_tick - Enable the scheduler tick on a CPU if RCU needs it. * * The scheduler tick is not normally enabled when CPUs enter the kernel * from nohz_full userspace execution. After all, nohz_full userspace * execution is an RCU quiescent state and the time executing in the kernel -* is quite short. Except of course when it isn't. And it is not hard to +* is quite short. Except of course when it isn't: it is not hard to * cause a large system to spend tens of seconds or even minutes looping -* in the kernel, which can cause a number of problems, include RCU CPU +* in the kernel, which can cause a number of problems, including RCU CPU * stall warnings. * * Therefore, if a nohz_full CPU fails to report a quiescent state @@ -879,7 +879,7 @@ void __rcu_irq_enter_check_tick(void) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data); -// Enabling the tick is unsafe in NMI handlers. +/* Enabling the tick is unsafe in NMI handlers. */ if (WARN_ON_ONCE (in_nmi ()))return ;@@ -889,21 +889,27 @@ void __rcu_irq_enter_check_tick(void)if (!tick_nohz_full_cpu (rdp->cpu) || !READ_ONCE (rdp->rcu_urgent_qs) ||READ_ONCE (rdp->rcu_forced_tick)) { -// RCU doesn't need nohz_full help from this CPU, or it is -// already getting that help. +/* +* RCU doesn't need nohz_full help from this CPU, or it is +* already getting that help. +*/ return ; } -// We get here only when not in an extended quiescent state and -// from interrupts (as opposed to NMIs). Therefore, (1) RCU is -// already watching and (2) The fact that we are in an interrupt -// handler and that the rcu_node lock is an irq-disabled lock -// prevents self-deadlock. So we can safely recheck under the lock. -// Note that the nohz_full state currently cannot change. +/* +* We get here only when not in an extended quiescent state and +* from interrupts (as opposed to NMIs). Therefore, (1) RCU is +* already watching and (2) the fact that we are in an interrupt +* handler and that the rcu_node lock is an irq-disabled lock +* prevents self-deadlock. So we can safely recheck under the lock. +* Note that the nohz_full state currently cannot change. +*/ raw_spin_lock_rcu_node (rdp->mynode);if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) { -// A nohz_full CPU is in the kernel and RCU needs a -// quiescent state. Turn on the tick! +/* +* A nohz_full CPU is in the kernel and RCU needs a +* quiescent state. Turn on the tick! +*/ WRITE_ONCE (rdp->rcu_forced_tick, true);tick_dep_set_cpu (rdp->cpu,TICK_DEP_BIT_RCU ); }
[tip: x86/entry] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()
In reply to Thomas GleixnerThe following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: aaf2bc50df1f4bfc6857fc601fc7b21d5a18c6a1 Gitweb: https://git.kernel.org/tip/aaf2bc50df1f4bfc6857fc601fc7b21d5a18c6a1 Author: Paul E. McKenney <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:15 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:04:18 +02:00
rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()
There will likely be exception handlers that can sleep, which rules out the usual approach of invoking rcu_nmi_enter() on entry and also rcu_nmi_exit() on all exit paths. However, the alternative approach of just not calling anything can prevent RCU from coaxing quiescent states from nohz_full CPUs that are looping in the kernel: RCU must instead IPI them explicitly. It would be better to enable the scheduler tick on such CPUs to interact with RCU in a lighter-weight manner, and this enabling is one of the things that rcu_nmi_enter() currently does.
What is needed is something that helps RCU coax quiescent states while not preventing subsequent sleeps. This commit therefore splits out the nohz_full scheduler-tick enabling from the rest of the rcu_nmi_enter() logic into a new function named rcu_irq_enter_check_tick().
[ tglx: Renamed the function and made it a nop when context tracking is off ] [ mingo: Fixed a CONFIG_NO_HZ_FULL assumption, harmonized and fixed all the
comment blocks and cleaned up rcu_nmi_enter()/exit() definitions. ]
--- include/linux/hardirq.h | 29 ++++++++------ kernel/rcu/tree.c | 82 ++++++++++++++++++++++++++++++---------- 2 files changed, 79 insertions(+), 32 deletions(-)
include/linux/hardirq.h +17 -12
@@ -2,31 +2,28 @@#ifndef LINUX_HARDIRQ_H #define LINUX_HARDIRQ_H +#include <linux/context_tracking_state.h> #include <linux/preempt.h> #include <linux/lockdep.h> #include <linux/ftrace_irq.h> #include <linux/vtime.h> #include <asm/hardirq.h> -extern void synchronize_irq (unsigned int irq);extern bool synchronize_hardirq (unsigned int irq); -#if defined(CONFIG_TINY_RCU ) - -static inline void rcu_nmi_enter (void ) -{ -} +#ifdef CONFIG_NO_HZ_FULL +void __rcu_irq_enter_check_tick (void ); +#else +static inline void __rcu_irq_enter_check_tick (void ) { } +#endif -static inline void rcu_nmi_exit (void ) +static __always_inline void rcu_irq_enter_check_tick(void) { +if (context_tracking_enabled ()) +__rcu_irq_enter_check_tick (); } -#else -extern void rcu_nmi_enter (void ); -extern void rcu_nmi_exit (void ); -#endif -/* * It is safe to do non-atomic ops on ->hardirq_context, * because NMI handlers may not preempt and the ops are
@@ -65,6 +62,14 @@ extern void irq_exit(void);#define arch_nmi_exit () do { } while (0)#endif +#ifdef CONFIG_TINY_RCU +static inline void rcu_nmi_enter (void ) { } +static inline void rcu_nmi_exit (void ) { } +#else +extern void rcu_nmi_enter (void ); +extern void rcu_nmi_exit (void ); +#endif +/* * NMI vs Tracing * --------------
kernel/rcu/tree.c +62 -20
94 lines not on this page · raw message
PATCH v9 3/39 rcu: Provide rcu_irq_exit_check_preempt()
In reply to Thomas GleixnerProvide a debug check which can be invoked from exception return to kernel mode before an attempt is made to schedule. Warn if RCU is not ready for this.
---
V9: New patch
--- include/linux/rcutiny.h | 1 + include/linux/rcutree.h | 6 ++++++ kernel/rcu/tree.c | 18 ++++++++++++++++++ 3 files changed, 25 insertions(+)
include/linux/rcutiny.h +1 -0
@@ -72,6 +72,7 @@ static inline void rcu_irq_exit_irqson(vstatic inline void rcu_irq_enter_irqson (void ) { }static inline void rcu_irq_exit (void ) { }static inline void rcu_irq_exit_preempt (void ) { } +static inline void rcu_irq_exit_check_preempt (void ) { }static inline void exit_rcu (void ) { }static inline bool rcu_preempt_need_deferred_qs (struct task_struct *t) {
include/linux/rcutree.h +6 -0
@@ -51,6 +51,12 @@ void rcu_irq_exit_preempt(void);void rcu_irq_enter_irqson (void );void rcu_irq_exit_irqson (void ); +#ifdef CONFIG_PROVE_RCU +void rcu_irq_exit_check_preempt (void ); +#else +static inline void rcu_irq_exit_check_preempt (void ) { } +#endif +void exit_rcu (void );void rcu_scheduler_starting (void );
kernel/rcu/tree.c +18 -0
@@ -765,6 +765,24 @@ void rcu_irq_exit_preempt(void) "RCU in extended quiescent state!"); } +#ifdef CONFIG_PROVE_RCU +/** +* rcu_irq_exit_check_preempt - Validate that scheduling is possible +*/ +void rcu_irq_exit_check_preempt (void ) +{ +lockdep_assert_irqs_disabled (); + +RCU_LOCKDEP_WARN (__this_cpu_read (rcu_data.dynticks_nesting) <=0 , +"RCU dynticks_nesting counter underflow/zero!" ); +RCU_LOCKDEP_WARN (__this_cpu_read (rcu_data.dynticks_nmi_nesting) != +DYNTICK_IRQ_NONIDLE , +"Bad RCU dynticks_nmi_nesting counter\n" ); +RCU_LOCKDEP_WARN (rcu_dynticks_curr_cpu_in_eqs (), +"RCU in extended quiescent state!" ); +} +#endif /* #ifdef CONFIG_PROVE_RCU */ +/* * Wrapper for rcu_irq_exit() where interrupts are enabled. *
[tip: x86/entry] rcu: Provide rcu_irq_exit_check_preempt()
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 07325d4a90d2d84de45cc07b134fd0f023dbb971 Gitweb: https://git.kernel.org/tip/07325d4a90d2d84de45cc07b134fd0f023dbb971 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:16 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:05:11 +02:00
rcu: Provide rcu_irq_exit_check_preempt()
Provide a debug check which can be invoked from exception return to kernel mode before an attempt is made to schedule. Warn if RCU is not ready for this.
--- include/linux/rcutiny.h | 1 + include/linux/rcutree.h | 6 ++++++ kernel/rcu/tree.c | 18 ++++++++++++++++++ 3 files changed, 25 insertions(+)
include/linux/rcutiny.h +1 -0
@@ -72,6 +72,7 @@ static inline void rcu_irq_exit_irqson(void) { }static inline void rcu_irq_enter_irqson (void ) { }static inline void rcu_irq_exit (void ) { }static inline void rcu_irq_exit_preempt (void ) { } +static inline void rcu_irq_exit_check_preempt (void ) { }static inline void exit_rcu (void ) { }static inline bool rcu_preempt_need_deferred_qs (struct task_struct *t) {
include/linux/rcutree.h +6 -0
@@ -51,6 +51,12 @@ void rcu_irq_exit_preempt(void);void rcu_irq_enter_irqson (void );void rcu_irq_exit_irqson (void ); +#ifdef CONFIG_PROVE_RCU +void rcu_irq_exit_check_preempt (void ); +#else +static inline void rcu_irq_exit_check_preempt (void ) { } +#endif +void exit_rcu (void );void rcu_scheduler_starting (void );
kernel/rcu/tree.c +18 -0
@@ -765,6 +765,24 @@ void rcu_irq_exit_preempt(void) "RCU in extended quiescent state!"); } +#ifdef CONFIG_PROVE_RCU +/** +* rcu_irq_exit_check_preempt - Validate that scheduling is possible +*/ +void rcu_irq_exit_check_preempt (void ) +{ +lockdep_assert_irqs_disabled (); + +RCU_LOCKDEP_WARN (__this_cpu_read (rcu_data.dynticks_nesting) <=0 , +"RCU dynticks_nesting counter underflow/zero!" ); +RCU_LOCKDEP_WARN (__this_cpu_read (rcu_data.dynticks_nmi_nesting) != +DYNTICK_IRQ_NONIDLE , +"Bad RCU dynticks_nmi_nesting counter\n" ); +RCU_LOCKDEP_WARN (rcu_dynticks_curr_cpu_in_eqs (), +"RCU in extended quiescent state!" ); +} +#endif /* #ifdef CONFIG_PROVE_RCU */ +/* * Wrapper for rcu_irq_exit() where interrupts are enabled. *
PATCH v9 4/39 x86/entry: Provide idtentry_entry/exit_cond_rcu()
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
After a lengthy discussion [1] it turned out that RCU does not need a full rcu_irq_enter/exit() when RCU is already watching. All it needs if NOHZ_FULL is active is to check whether the tick needs to be restarted.
This allows to avoid a separate variant for the pagefault handler which cannot invoke rcu_irq_enter() on a kernel pagefault which might sleep.
The cond_rcu argument is only temporary and will be removed once the existing users of idtentry_enter/exit() have been cleaned up. After that the code can be significantly simplified.
---
V9: Reworked to the new RCU mode
--- arch/x86/entry/common.c | 84 ++++++++++++++++++++++++++++++---------- arch/x86/include/asm/idtentry.h | 14 +++++- 2 files changed, 77 insertions(+), 21 deletions(-)
arch/x86/entry/common.c +65 -19
@@ -512,8 +512,10 @@ SYSCALL_DEFINE0(ni_syscall) }/** -* idtentry_enter - Handle state tracking on idtentry +* idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional +* RCU handling * @regs: Pointer to pt_regs of interrupted context +* @cond_rcu: Invoke rcu_irq_enter() only if RCU is not watching * * Invokes: * - lockdep irqflag state tracking as low level ASM entry disabled
@@ -521,40 +523,81 @@ SYSCALL_DEFINE0(ni_syscall)* * - Context tracking if the exception hit user mode. * -* - RCU notification if the exception hit kernel mode. -* * - The hardirq tracer to keep the state consistent as low level ASM * entry disabled interrupts. +* +* For kernel mode entries RCU handling is done conditional. If RCU is +* watching then the only RCU requirement is to check whether the tick has +* to be restarted. If RCU is not watching then rcu_irq_enter() has to be +* invoked on entry and rcu_irq_exit() on exit. +* +* Avoiding the rcu_irq_enter/exit() calls is an optimization but also +* solves the problem of kernel mode pagefaults which can schedule, which +* is not possible after invoking rcu_irq_enter() without undoing it. +* +* For user mode entries enter_from_user_mode() must be invoked to +* establish the proper context for NOHZ_FULL. Otherwise scheduling on exit +* would not be possible. +* +* Returns: True if RCU has been adjusted on a kernel entry +* False otherwise +* +* The return value must be fed into the rcu_exit argument of +* idtentry_exit_cond_rcu(). */ -void noinstr idtentry_enter(struct pt_regs *regs) +bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu) {if (user_mode (regs)) {enter_from_user_mode (); }else { -lockdep_hardirqs_off (CALLER_ADDR0 ); -rcu_irq_enter (); -instrumentation_begin (); -trace_hardirqs_off_prepare (); -instrumentation_end (); +if (!cond_rcu || !__rcu_is_watching ()) { +/* +* If RCU is not watching then the same careful +* sequence vs. lockdep and tracing is required +* as in enter_from_user_mode(). +* +* This only happens for IRQs that hits the idle +* loop, i.e. if idle is not using MWAIT. +*/ +lockdep_hardirqs_off (CALLER_ADDR0 ); +rcu_irq_enter (); +instrumentation_begin (); +trace_hardirqs_off_prepare (); +instrumentation_end (); +return true; + }else { +/* +* If RCU is watching then RCU only wants to check +* whether it needs to restart the tick in NOHZ +* mode. +*/ +instrumentation_begin (); +rcu_irq_enter_check_tick (); +/* Use the combo lockdep/tracing function */ +trace_hardirqs_off (); +instrumentation_end (); + } } +return false; }/** -* idtentry_exit - Common code to handle return from exceptions +* idtentry_exit_cond_rcu - Handle return from exception with conditional RCU +* handling * @regs: Pointer to pt_regs (exception entry regs) +* @rcu_exit: Invoke rcu_irq_exit() if true * * Depending on the return target (kernel/user) this runs the necessary -* preemption and work checks if possible and required and returns to +* preemption and work checks if possible and reguired and returns to * the caller with interrupts disabled and no further work pending. * * This is the last action before returning to the low level ASM code which * just needs to return to the appropriate context. * -* Invoked by all exception/interrupt IDTENTRY handlers which are not -* returning through the paranoid exit path (all except NMI, #DF and the IST -* variants of #MC and #DB) and are therefore on the thread stack. +* Counterpart to idtentry_enter_cond_rcu(). The return value of the entry +* function must be fed into the @rcu_exit argument. */ -void noinstr idtentry_exit(struct pt_regs *regs) +void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit) {lockdep_assert_irqs_disabled ();
@@ -580,7 +623,8 @@ void noinstr idtentry_exit(struct pt_regif (IS_ENABLED (CONFIG_DEBUG_ENTRY ))WARN_ON_ONCE (!on_thread_stack ());instrumentation_begin (); -rcu_irq_exit_preempt (); +if (rcu_exit) +rcu_irq_exit_preempt ();if (need_resched ())preempt_schedule_irq ();/* Covers both tracing and lockdep */
@@ -602,10 +646,12 @@ void noinstr idtentry_exit(struct pt_regtrace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (CALLER_ADDR0 );instrumentation_end (); -rcu_irq_exit (); +if (rcu_exit) +rcu_irq_exit ();lockdep_hardirqs_on (CALLER_ADDR0 ); }else { -/* IRQ flags state is correct already. Just tell RCU */ -rcu_irq_exit (); +/* IRQ flags state is correct already. Just tell RCU. */ +if (rcu_exit) +rcu_irq_exit (); } }
arch/x86/include/asm/idtentry.h +12 -2
@@ -7,8 +7,18 @@#ifndef __ASSEMBLY__ -void idtentry_enter (struct pt_regs *regs); -void idtentry_exit (struct pt_regs *regs); +bool idtentry_enter_cond_rcu (struct pt_regs *regs,bool cond_rcu); +void idtentry_exit_cond_rcu (struct pt_regs *regs,bool rcu_exit); + +static __always_inline void idtentry_enter(struct pt_regs *regs) +{ +idtentry_enter_cond_rcu (regs, false); +} + +static __always_inline void idtentry_exit(struct pt_regs *regs) +{ +idtentry_exit_cond_rcu (regs, true); +}/** * DECLARE_IDTENTRY - Declare functions for simple IDT entry points
Paul E. McKenney
maintainerRe: [patch V9 04/39] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 6 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
After a lengthy discussion [1] it turned out that RCU does not need a full rcu_irq_enter/exit() when RCU is already watching. All it needs if
NOHZ_FULLis active is to check whether the tick needs to be restarted.
Quoted 6 lines
This allows to avoid a separate variant for the pagefault handler which cannot invoke
rcu_irq_enter()on a kernel pagefault which might sleep.The
cond_rcuargument is only temporary and will be removed once the existing users of idtentry_enter/exit() have been cleaned up. After that the code can be significantly simplified.
And yes, I see that the sole call to rcu_irq_enter_check_tick() does have instrumentation turned on, so good! ;-)
Thanx, Paul
Re: [patch V9 04/39] x86/entry: Provide idtentry_entry/exit_cond_rcu()
In reply to Thomas GleixnerQuoted 95 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
After a lengthy discussion [1] it turned out that RCU does not need a full rcu_irq_enter/exit() when RCU is already watching. All it needs if
NOHZ_FULLis active is to check whether the tick needs to be restarted.This allows to avoid a separate variant for the pagefault handler which cannot invoke
rcu_irq_enter()on a kernel pagefault which might sleep.The
cond_rcuargument is only temporary and will be removed once the existing users of idtentry_enter/exit() have been cleaned up. After that the code can be significantly simplified.Signed-off-by Thomas Gleixner <[email protected]>---
V9: Reworked to the new RCU mode--- arch/x86/entry/common.c | 84 ++++++++++++++++++++++++++++++---------- arch/x86/include/asm/idtentry.h | 14 +++++- 2 files changed, 77 insertions(+), 21 deletions(-)arch/x86/entry/common.c +43
-9@@ -512,8 +512,10 @@ SYSCALL_DEFINE0(ni_syscall) }/** -* idtentry_enter - Handle state tracking on idtentry +* idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional +* RCU handling * @regs: Pointer to pt_regs of interrupted context +* @cond_rcu: Invoke rcu_irq_enter() only if RCU is not watching * * Invokes: * - lockdep irqflag state tracking as low level ASM entry disabled @@ -521,40 +523,81 @@ SYSCALL_DEFINE0(ni_syscall)* * - Context tracking if the exception hit user mode. * -* - RCU notification if the exception hit kernel mode. -* * - The hardirq tracer to keep the state consistent as low level ASM * entry disabled interrupts. +* +* For kernel mode entries RCU handling is done conditional. If RCU is +* watching then the only RCU requirement is to check whether the tick has +* to be restarted. If RCU is not watching then rcu_irq_enter() has to be +* invoked on entry and rcu_irq_exit() on exit. +* +* Avoiding the rcu_irq_enter/exit() calls is an optimization but also +* solves the problem of kernel mode pagefaults which can schedule, which +* is not possible after invoking rcu_irq_enter() without undoing it. +* +* For user mode entries enter_from_user_mode() must be invoked to +* establish the proper context for NOHZ_FULL. Otherwise scheduling on exit +* would not be possible. +* +* Returns: True if RCU has been adjusted on a kernel entry +* False otherwise +* +* The return value must be fed into the rcu_exit argument of +* idtentry_exit_cond_rcu(). */ -void noinstr idtentry_enter(struct pt_regs *regs) +bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu) {if (user_mode (regs)) {enter_from_user_mode (); }else { -lockdep_hardirqs_off (CALLER_ADDR0 ); -rcu_irq_enter (); -instrumentation_begin (); -trace_hardirqs_off_prepare (); -instrumentation_end (); +if (!cond_rcu || !__rcu_is_watching ()) { +/* +* If RCU is not watching then the same careful +* sequence vs. lockdep and tracing is required +* as in enter_from_user_mode(). +* +* This only happens for IRQs that hits the idle +* loop, i.e. if idle is not using MWAIT. +*/ +lockdep_hardirqs_off (CALLER_ADDR0 ); +rcu_irq_enter (); +instrumentation_begin (); +trace_hardirqs_off_prepare (); +instrumentation_end (); +return true; + }else { +/* +* If RCU is watching then RCU only wants to check +* whether it needs to restart the tick in NOHZ +* mode.
arch/x86/entry/common.c SYSCALL_DEFINE0(ni_syscall) · line 582
Quoted 10 lines
arch/x86/entry/common.c +8
-0+*/ +instrumentation_begin (); +rcu_irq_enter_check_tick (); +/* Use the combo lockdep/tracing function */ +trace_hardirqs_off (); +instrumentation_end (); + } } +return false; }
Any objections to the simplified/flattened control flow below?
Thanks,
Ingo
common.c | 59 +++++++++++++++++++++++++++++++---------------------------- 1 file changed, 31 insertions(+), 28 deletions(-)
arch/x86/entry/common.c +31 -28
@@ -549,35 +549,38 @@ bool noinstr idtentry_enter_cond_rcu(str {if (user_mode (regs)) {enter_from_user_mode (); - }else { -if (!cond_rcu || !__rcu_is_watching ()) { -/* -* If RCU is not watching then the same careful -* sequence vs. lockdep and tracing is required -* as in enter_from_user_mode(). -* -* This only happens for IRQs that hit the idle -* loop, i.e. if idle is not using MWAIT. -*/ -lockdep_hardirqs_off (CALLER_ADDR0 ); -rcu_irq_enter (); -instrumentation_begin (); -trace_hardirqs_off_prepare (); -instrumentation_end (); -return true; - }else { -/* -* If RCU is watching then RCU only wants to check -* whether it needs to restart the tick in NOHZ -* mode. -*/ -instrumentation_begin (); -rcu_irq_enter_check_tick (); -/* Use the combo lockdep/tracing function */ -trace_hardirqs_off (); -instrumentation_end (); - } +return false; } + +if (!cond_rcu || !__rcu_is_watching ()) { +/* +* If RCU is not watching then the same careful +* sequence vs. lockdep and tracing is required +* as in enter_from_user_mode(). +* +* This only happens for IRQs that hit the idle +* loop, i.e. if idle is not using MWAIT. +*/ +lockdep_hardirqs_off (CALLER_ADDR0 ); +rcu_irq_enter (); +instrumentation_begin (); +trace_hardirqs_off_prepare (); +instrumentation_end (); + +return true; + } + +/* +* If RCU is watching then RCU only wants to check +* whether it needs to restart the tick in NOHZ +* mode. +*/ +instrumentation_begin (); +rcu_irq_enter_check_tick (); +/* Use the combo lockdep/tracing function */ +trace_hardirqs_off (); +instrumentation_end (); +return false; }
Re: [patch V9 04/39] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 3 lines · Ingo Molnar
* Thomas Gleixner <[email protected]> wrote:
Any objections to the simplified/flattened control flow below?
No, but you really don't want to do it at this point. Patch 7/39 is where you want to do that.
PATCH v9 5/39 x86/entry: Provide idtentry_enter/exit_user()
In reply to Thomas GleixnerAs there are exceptions which already handle entry from user mode and from kernel mode separately having an explicit user handling makes sense and makes the code easier to understand.
---
V9: New patch
--- arch/x86/entry/common.c | 31 +++++++++++++++++++++++++++++++ arch/x86/include/asm/idtentry.h | 3 +++ 2 files changed, 34 insertions(+)
arch/x86/entry/common.c +31 -0
@@ -655,3 +655,34 @@ void noinstr idtentry_exit_cond_rcu(strurcu_irq_exit (); } } + +/** +* idtentry_enter_user - Handle state tracking on idtentry from user mode +* @regs: Pointer to pt_regs of interrupted context +* +* Invokes enter_from_user_mode() to establish the proper context for +* NOHZ_FULL. Otherwise scheduling on exit would not be possible. +*/ +void noinstr idtentry_enter_user(struct pt_regs *regs) +{ +enter_from_user_mode (); +} + +/** +* idtentry_exit_user - Handle return from exception to user mode +* @regs: Pointer to pt_regs (exception entry regs) +* +* Runs the necessary preemption and work checks and returns to the caller +* with interrupts disabled and no further work pending. +* +* This is the last action before returning to the low level ASM code which +* just needs to return to the appropriate context. +* +* Counterpart to idtentry_enter_user(). +*/ +void noinstr idtentry_exit_user(struct pt_regs *regs) +{ +lockdep_assert_irqs_disabled (); + +prepare_exit_to_usermode (regs); +}
arch/x86/include/asm/idtentry.h +3 -0
@@ -7,6 +7,9 @@#ifndef __ASSEMBLY__ +void idtentry_enter_user (struct pt_regs *regs); +void idtentry_exit_user (struct pt_regs *regs); +bool idtentry_enter_cond_rcu (struct pt_regs *regs,bool cond_rcu);void idtentry_exit_cond_rcu (struct pt_regs *regs,bool rcu_exit);
[tip: x86/entry] x86/entry: Provide idtentry_enter/exit_user()
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: d4b1f51731fd10a836e46f54c96a8913d945735b Gitweb: https://git.kernel.org/tip/d4b1f51731fd10a836e46f54c96a8913d945735b Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:18 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
x86/entry: Provide idtentry_enter/exit_user()
As there are exceptions which already handle entry from user mode and from kernel mode separately, providing explicit user entry/exit handling callbacks makes sense and makes the code easier to understand.
--- arch/x86/entry/common.c | 31 +++++++++++++++++++++++++++++++ arch/x86/include/asm/idtentry.h | 3 +++ 2 files changed, 34 insertions(+)
arch/x86/entry/common.c +31 -0
@@ -658,3 +658,34 @@ void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit)rcu_irq_exit (); } } + +/** +* idtentry_enter_user - Handle state tracking on idtentry from user mode +* @regs: Pointer to pt_regs of interrupted context +* +* Invokes enter_from_user_mode() to establish the proper context for +* NOHZ_FULL. Otherwise scheduling on exit would not be possible. +*/ +void noinstr idtentry_enter_user(struct pt_regs *regs) +{ +enter_from_user_mode (); +} + +/** +* idtentry_exit_user - Handle return from exception to user mode +* @regs: Pointer to pt_regs (exception entry regs) +* +* Runs the necessary preemption and work checks and returns to the caller +* with interrupts disabled and no further work pending. +* +* This is the last action before returning to the low level ASM code which +* just needs to return to the appropriate context. +* +* Counterpart to idtentry_enter_user(). +*/ +void noinstr idtentry_exit_user(struct pt_regs *regs) +{ +lockdep_assert_irqs_disabled (); + +prepare_exit_to_usermode (regs); +}
arch/x86/include/asm/idtentry.h +3 -0
@@ -7,6 +7,9 @@#ifndef __ASSEMBLY__ +void idtentry_enter_user (struct pt_regs *regs); +void idtentry_exit_user (struct pt_regs *regs); +bool idtentry_enter_cond_rcu (struct pt_regs *regs,bool cond_rcu);void idtentry_exit_cond_rcu (struct pt_regs *regs,bool rcu_exit);
PATCH v9 6/39 x86/idtentry: Switch to conditional RCU handling
In reply to Thomas GleixnerSwitch all idtentry_enter/exit() users over to the new conditional RCU handling scheme and make the user mode entries in #DB, #INT3 and #MCE use the user mode idtentry functions.
---
V9: New patch
--- arch/x86/include/asm/idtentry.h | 10 ++++++---- arch/x86/kernel/cpu/mce/core.c | 4 ++-- arch/x86/kernel/traps.c | 10 +++++----- 3 files changed, 13 insertions(+), 11 deletions(-)
arch/x86/include/asm/idtentry.h +6 -4
@@ -61,11 +61,12 @@ static __always_inline void __##func(str \ __visible noinstr void func(struct pt_regs *regs) \ { \ -idtentry_enter (regs); \ +bool rcu_exit =idtentry_enter_cond_rcu (regs); \ + \instrumentation_begin (); \ __##func (regs); \ instrumentation_end(); \ - idtentry_exit(regs); \ + idtentry_exit_cond_rcu(regs, rcu_exit); \ } \ \ static __always_inline void __##func(struct pt_regs *regs)
@@ -107,11 +108,12 @@ static __always_inline void __##func(str __visible noinstr void func(struct pt_regs *regs, \unsigned long error_code) \ { \ -idtentry_enter (regs); \ +bool rcu_exit =idtentry_enter_cond_rcu (regs); \ + \instrumentation_begin (); \ __##func (regs, error_code); \ instrumentation_end(); \ - idtentry_exit(regs); \ + idtentry_exit_cond_rcu(regs, rcu_exit); \ } \ \ static __always_inline void __##func(struct pt_regs *regs, \
arch/x86/kernel/cpu/mce/core.c +2 -2
@@ -1929,11 +1929,11 @@ static __always_inline void exc_machine_ static __always_inline void exc_machine_check_user(struct pt_regs *regs) { -idtentry_enter (regs); +idtentry_enter_user (regs);instrumentation_begin ();machine_check_vector (regs);instrumentation_end (); -idtentry_exit (regs); +idtentry_exit_user (regs); }#ifdef CONFIG_X86_64
arch/x86/kernel/traps.c +5 -5
@@ -619,18 +619,18 @@ DEFINE_IDTENTRY_RAW(exc_int3)return ;/* -* idtentry_enter() uses static_branch_{,un}likely() and therefore +* idtentry_enter_user() uses static_branch_{,un}likely() and therefore * can trigger INT3, hence poke_int3_handler() must be done * before. If the entry came from kernel mode, then use nmi_enter() * because the INT3 could have been hit in any context including * NMI. */ if (user_mode (regs)) { -idtentry_enter (regs); +idtentry_enter_user (regs);instrumentation_begin ();do_int3_user (regs);instrumentation_end (); -idtentry_exit (regs); +idtentry_exit_user (regs); }else {nmi_enter ();instrumentation_begin ();
@@ -877,7 +877,7 @@ static __always_inline void exc_debug_ke static __always_inline void exc_debug_user(struct pt_regs *regs,unsigned long dr6) { -idtentry_enter (regs); +idtentry_enter_user (regs);clear_thread_flag (TIF_BLOCKSTEP );/*
@@ -886,7 +886,7 @@ static __always_inline void exc_debug_us* User wants a sigtrap for that. */ handle_debug (regs, dr6, !dr6); -idtentry_exit (regs); +idtentry_exit_user (regs); }#ifdef CONFIG_X86_64
[tip: x86/entry] x86/idtentry: Switch to conditional RCU handling
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 77d7843297dfe3bbd997a46548c9cdd8047ecffe Gitweb: https://git.kernel.org/tip/77d7843297dfe3bbd997a46548c9cdd8047ecffe Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:19 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
x86/idtentry: Switch to conditional RCU handling
Switch all idtentry_enter/exit() users over to the new conditional RCU handling scheme and make the user mode entries in #DB, #INT3 and #MCE use the user mode idtentry functions.
--- arch/x86/include/asm/idtentry.h | 10 ++++++---- arch/x86/kernel/cpu/mce/core.c | 4 ++-- arch/x86/kernel/traps.c | 10 +++++----- 3 files changed, 13 insertions(+), 11 deletions(-)
arch/x86/include/asm/idtentry.h +6 -4
@@ -61,11 +61,12 @@ static __always_inline void __##func(struct pt_regs *regs); \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ -idtentry_enter (regs); \ +bool rcu_exit =idtentry_enter_cond_rcu (regs); \ + \instrumentation_begin (); \ __##func (regs); \ instrumentation_end(); \ - idtentry_exit(regs); \ + idtentry_exit_cond_rcu(regs, rcu_exit); \ } \ \ static __always_inline void __##func(struct pt_regs *regs)
@@ -107,11 +108,12 @@ static __always_inline void __##func(struct pt_regs *regs, \ __visible noinstr void func(struct pt_regs *regs, \unsigned long error_code) \ { \ -idtentry_enter (regs); \ +bool rcu_exit =idtentry_enter_cond_rcu (regs); \ + \instrumentation_begin (); \ __##func (regs, error_code); \ instrumentation_end(); \ - idtentry_exit(regs); \ + idtentry_exit_cond_rcu(regs, rcu_exit); \ } \ \ static __always_inline void __##func(struct pt_regs *regs, \
arch/x86/kernel/cpu/mce/core.c +2 -2
@@ -1929,11 +1929,11 @@ static __always_inline void exc_machine_check_kernel(struct pt_regs *regs) static __always_inline void exc_machine_check_user(struct pt_regs *regs) { -idtentry_enter (regs); +idtentry_enter_user (regs);instrumentation_begin ();machine_check_vector (regs);instrumentation_end (); -idtentry_exit (regs); +idtentry_exit_user (regs); }#ifdef CONFIG_X86_64
arch/x86/kernel/traps.c +5 -5
@@ -619,18 +619,18 @@ DEFINE_IDTENTRY_RAW(exc_int3)return ;/* -* idtentry_enter() uses static_branch_{,un}likely() and therefore +* idtentry_enter_user() uses static_branch_{,un}likely() and therefore * can trigger INT3, hence poke_int3_handler() must be done * before. If the entry came from kernel mode, then use nmi_enter() * because the INT3 could have been hit in any context including * NMI. */ if (user_mode (regs)) { -idtentry_enter (regs); +idtentry_enter_user (regs);instrumentation_begin ();do_int3_user (regs);instrumentation_end (); -idtentry_exit (regs); +idtentry_exit_user (regs); }else {nmi_enter ();instrumentation_begin ();
@@ -877,7 +877,7 @@ static __always_inline void exc_debug_kernel(struct pt_regs *regs, static __always_inline void exc_debug_user(struct pt_regs *regs,unsigned long dr6) { -idtentry_enter (regs); +idtentry_enter_user (regs);clear_thread_flag (TIF_BLOCKSTEP );/*
@@ -886,7 +886,7 @@ static __always_inline void exc_debug_user(struct pt_regs *regs,* User wants a sigtrap for that. */ handle_debug (regs, dr6, !dr6); -idtentry_exit (regs); +idtentry_exit_user (regs); }#ifdef CONFIG_X86_64
PATCH v9 7/39 x86/entry: Cleanup idtentry_enter/exit() leftovers
In reply to Thomas GleixnerNow that everything is converted to conditional RCU handling remove idtentry_enter/exit() and tidy up the conditional functions.
This does not remove rcu_irq_exit_preempt() to avoid conflicts with the RCU tree. Will be removed once all of this hit Linus tree.
---
V9: New patch
--- arch/x86/entry/common.c | 67 +++++++++++++++++----------------------- arch/x86/include/asm/idtentry.h | 12 ------- 2 files changed, 30 insertions(+), 49 deletions(-)
arch/x86/entry/common.c +29 -38
@@ -515,7 +515,6 @@ SYSCALL_DEFINE0(ni_syscall)* idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional * RCU handling * @regs: Pointer to pt_regs of interrupted context -* @cond_rcu: Invoke rcu_irq_enter() only if RCU is not watching * * Invokes: * - lockdep irqflag state tracking as low level ASM entry disabled
@@ -545,12 +544,12 @@ SYSCALL_DEFINE0(ni_syscall)* The return value must be fed into the rcu_exit argument of * idtentry_exit_cond_rcu(). */ -bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu) +bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs) {if (user_mode (regs)) {enter_from_user_mode (); }else { -if (!cond_rcu || !__rcu_is_watching ()) { +if (!__rcu_is_watching ()) {/* * If RCU is not watching then the same careful * sequence vs. lockdep and tracing is required
@@ -605,52 +604,44 @@ void noinstr idtentry_exit_cond_rcu(struif (user_mode (regs)) {prepare_exit_to_usermode (regs); }else if (regs->flags &X86_EFLAGS_IF ) { +/* +* If RCU was not watching on entry this needs to be done +* carefully and needs the same ordering of lockdep/tracing +* and RCU as the return to user mode path. +*/ +if (rcu_exit) { +instrumentation_begin (); +/* Tell the tracer that IRET will enable interrupts */ +trace_hardirqs_on_prepare (); +lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); +instrumentation_end (); +rcu_irq_exit (); +lockdep_hardirqs_on (CALLER_ADDR0 ); +return ; + } + +instrumentation_begin (); +/* Check kernel preemption, if enabled */ if (IS_ENABLED (CONFIG_PREEMPTION )) { -/* -* This needs to be done very carefully. -* idtentry_enter() invoked rcu_irq_enter(). This -* needs to be undone before scheduling. -* -* Preemption is disabled inside of RCU idle -* sections. When the task returns from -* preempt_schedule_irq(), RCU is still watching. -* -* rcu_irq_exit_preempt() has additional state -* checking if CONFIG_PROVE_RCU=y -*/ if (!preempt_count ()) { +/* Sanity check RCU and thread stack */ +rcu_irq_exit_check_preempt ();if (IS_ENABLED (CONFIG_DEBUG_ENTRY ))WARN_ON_ONCE (!on_thread_stack ()); -instrumentation_begin (); -if (rcu_exit) -rcu_irq_exit_preempt ();if (need_resched ())preempt_schedule_irq (); -/* Covers both tracing and lockdep */ -trace_hardirqs_on (); -instrumentation_end (); -return ; } } -/* -* If preemption is disabled then this needs to be done -* carefully with respect to RCU. The exception might come -* from a RCU idle section in the idle task due to the fact -* that safe_halt() enables interrupts. So this needs the -* same ordering of lockdep/tracing and RCU as the return -* to user mode path. -*/ -instrumentation_begin (); -/* Tell the tracer that IRET will enable interrupts */ -trace_hardirqs_on_prepare (); -lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); +/* Covers both tracing and lockdep */ +trace_hardirqs_on (); +instrumentation_end (); -if (rcu_exit) -rcu_irq_exit (); -lockdep_hardirqs_on (CALLER_ADDR0 ); }else { -/* IRQ flags state is correct already. Just tell RCU. */ +/* +* IRQ flags state is correct already. Just tell RCU if it +* was not watching on entry. +*/ if (rcu_exit)rcu_irq_exit (); }
arch/x86/include/asm/idtentry.h +1 -11
@@ -10,19 +10,9 @@void idtentry_enter_user (struct pt_regs *regs);void idtentry_exit_user (struct pt_regs *regs); -bool idtentry_enter_cond_rcu (struct pt_regs *regs,bool cond_rcu); +bool idtentry_enter_cond_rcu (struct pt_regs *regs);void idtentry_exit_cond_rcu (struct pt_regs *regs,bool rcu_exit); -static __always_inline void idtentry_enter(struct pt_regs *regs) -{ -idtentry_enter_cond_rcu (regs, false); -} - -static __always_inline void idtentry_exit(struct pt_regs *regs) -{ -idtentry_exit_cond_rcu (regs, true); -} -/** * DECLARE_IDTENTRY - Declare functions for simple IDT entry points * No error code pushed by hardware
[tip: x86/entry] x86/entry: Clean up idtentry_enter/exit() leftovers
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 1951bc04047a0ca5c88146c4b19b296d40e489ef Gitweb: https://git.kernel.org/tip/1951bc04047a0ca5c88146c4b19b296d40e489ef Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:20 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
x86/entry: Clean up idtentry_enter/exit() leftovers
Now that everything is converted to conditional RCU handling remove idtentry_enter/exit() and tidy up the conditional functions.
This does not remove rcu_irq_exit_preempt(), to avoid conflicts with the RCU tree. Will be removed once all of this hits Linus's tree.
--- arch/x86/entry/common.c | 67 +++++++++++++------------------- arch/x86/include/asm/idtentry.h | 12 +------ 2 files changed, 30 insertions(+), 49 deletions(-)
arch/x86/entry/common.c +29 -38
@@ -515,7 +515,6 @@ SYSCALL_DEFINE0(ni_syscall)* idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional * RCU handling * @regs: Pointer to pt_regs of interrupted context -* @cond_rcu: Invoke rcu_irq_enter() only if RCU is not watching * * Invokes: * - lockdep irqflag state tracking as low level ASM entry disabled
@@ -545,14 +544,14 @@ SYSCALL_DEFINE0(ni_syscall)* The return value must be fed into the rcu_exit argument of * idtentry_exit_cond_rcu(). */ -bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu) +bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs) {if (user_mode (regs)) {enter_from_user_mode ();return false; } -if (!cond_rcu || !__rcu_is_watching ()) { +if (!__rcu_is_watching ()) {/* * If RCU is not watching then the same careful * sequence vs. lockdep and tracing is required
@@ -608,52 +607,44 @@ void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit)if (user_mode (regs)) {prepare_exit_to_usermode (regs); }else if (regs->flags &X86_EFLAGS_IF ) { +/* +* If RCU was not watching on entry this needs to be done +* carefully and needs the same ordering of lockdep/tracing +* and RCU as the return to user mode path. +*/ +if (rcu_exit) { +instrumentation_begin (); +/* Tell the tracer that IRET will enable interrupts */ +trace_hardirqs_on_prepare (); +lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); +instrumentation_end (); +rcu_irq_exit (); +lockdep_hardirqs_on (CALLER_ADDR0 ); +return ; + } + +instrumentation_begin (); +/* Check kernel preemption, if enabled */ if (IS_ENABLED (CONFIG_PREEMPTION )) { -/* -* This needs to be done very carefully. -* idtentry_enter() invoked rcu_irq_enter(). This -* needs to be undone before scheduling. -* -* Preemption is disabled inside of RCU idle -* sections. When the task returns from -* preempt_schedule_irq(), RCU is still watching. -* -* rcu_irq_exit_preempt() has additional state -* checking if CONFIG_PROVE_RCU=y -*/ if (!preempt_count ()) { +/* Sanity check RCU and thread stack */ +rcu_irq_exit_check_preempt ();if (IS_ENABLED (CONFIG_DEBUG_ENTRY ))WARN_ON_ONCE (!on_thread_stack ()); -instrumentation_begin (); -if (rcu_exit) -rcu_irq_exit_preempt ();if (need_resched ())preempt_schedule_irq (); -/* Covers both tracing and lockdep */ -trace_hardirqs_on (); -instrumentation_end (); -return ; } } -/* -* If preemption is disabled then this needs to be done -* carefully with respect to RCU. The exception might come -* from a RCU idle section in the idle task due to the fact -* that safe_halt() enables interrupts. So this needs the -* same ordering of lockdep/tracing and RCU as the return -* to user mode path. -*/ -instrumentation_begin (); -/* Tell the tracer that IRET will enable interrupts */ -trace_hardirqs_on_prepare (); -lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); +/* Covers both tracing and lockdep */ +trace_hardirqs_on (); +instrumentation_end (); -if (rcu_exit) -rcu_irq_exit (); -lockdep_hardirqs_on (CALLER_ADDR0 ); }else { -/* IRQ flags state is correct already. Just tell RCU. */ +/* +* IRQ flags state is correct already. Just tell RCU if it +* was not watching on entry. +*/ if (rcu_exit)rcu_irq_exit (); }
arch/x86/include/asm/idtentry.h +1 -11
@@ -10,19 +10,9 @@void idtentry_enter_user (struct pt_regs *regs);void idtentry_exit_user (struct pt_regs *regs); -bool idtentry_enter_cond_rcu (struct pt_regs *regs,bool cond_rcu); +bool idtentry_enter_cond_rcu (struct pt_regs *regs);void idtentry_exit_cond_rcu (struct pt_regs *regs,bool rcu_exit); -static __always_inline void idtentry_enter(struct pt_regs *regs) -{ -idtentry_enter_cond_rcu (regs, false); -} - -static __always_inline void idtentry_exit(struct pt_regs *regs) -{ -idtentry_exit_cond_rcu (regs, true); -} -/** * DECLARE_IDTENTRY - Declare functions for simple IDT entry points * No error code pushed by hardware
PATCH v9 8/39 genirq: Provide irq_enter/exit_rcu()
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
irq_enter()/exit() include the RCU handling. To properly separate the RCU handling provide variants which contain only the non-RCU related functionality.
--- include/linux/hardirq.h | 13 +++++++++++-- kernel/softirq.c | 35 +++++++++++++++++++++++++++-------- 2 files changed, 38 insertions(+), 10 deletions(-)
include/linux/hardirq.h +11 -2
@@ -52,7 +52,11 @@ static __always_inline void rcu_irq_ente/* * Enter irq context (on NO_HZ, update jiffies): */ -extern void irq_enter (void ); +void irq_enter (void ); +/* +* Like irq_enter(), but RCU is already watching. +*/ +void irq_enter_rcu (void );/* * Exit irq context without processing softirqs:
@@ -67,7 +71,12 @@ extern void irq_enter(void);/* * Exit irq context and process softirqs if needed: */ -extern void irq_exit (void ); +void irq_exit (void ); + +/* +* Like irq_exit(), but return with RCU watching. +*/ +void irq_exit_rcu (void );#ifndef arch_nmi_enter#define arch_nmi_enter () do { } while (0)
kernel/softirq.c +27 -8
@@ -339,12 +339,11 @@ asmlinkage __visible void do_softirq(voilocal_irq_restore (flags); } -/* -* Enter an interrupt context. +/** +* irq_enter_rcu - Enter an interrupt context with RCU watching */ -void irq_enter (void ) +void irq_enter_rcu (void ) { -rcu_irq_enter ();if (is_idle_task (current) && !in_interrupt ()) {/* * Prevent raise_softirq from needlessly waking up ksoftirqd
@@ -354,10 +353,18 @@ void irq_enter(void)tick_irq_enter ();_local_bh_enable (); } -__irq_enter (); } +/** +* irq_enter - Enter an interrupt context including RCU update +*/ +void irq_enter (void ) +{ +rcu_irq_enter (); +irq_enter_rcu (); +} +static inline void invoke_softirq (void ) {if (ksoftirqd_running (local_softirq_pending ()))
@@ -397,10 +404,12 @@ static inline void tick_irq_exit(void)#endif } -/* -* Exit an interrupt context. Process softirqs if needed and possible: +/** +* irq_exit_rcu() - Exit an interrupt context without updating RCU +* +* Also processes softirqs if needed and possible. */ -void irq_exit (void ) +void irq_exit_rcu (void ) {#ifndef __ARCH_IRQ_EXIT_IRQS_DISABLEDlocal_irq_disable ();
@@ -413,6 +422,16 @@ void irq_exit(void)invoke_softirq ();tick_irq_exit (); +} + +/** +* irq_exit - Exit an interrupt context, update RCU and lockdep +* +* Also processes softirqs if needed and possible. +*/ +void irq_exit (void ) +{ +irq_exit_rcu ();rcu_irq_exit ();/* must be last! */ lockdep_hardirq_exit ();
[tip: x86/entry] genirq: Provide irq_enter/exit_rcu()
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: a59262f6413b82992626754b46383a615bfac152 Gitweb: https://git.kernel.org/tip/a59262f6413b82992626754b46383a615bfac152 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:21 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
genirq: Provide irq_enter/exit_rcu()
irq_enter()/exit() currently include RCU handling. To properly separate the RCU handling code, provide variants which contain only the non-RCU related functionality.
--- include/linux/hardirq.h | 13 +++++++++++-- kernel/softirq.c | 35 +++++++++++++++++++++++++++-------- 2 files changed, 38 insertions(+), 10 deletions(-)
include/linux/hardirq.h +11 -2
@@ -40,7 +40,11 @@ static __always_inline void rcu_irq_enter_check_tick(void)/* * Enter irq context (on NO_HZ, update jiffies): */ -extern void irq_enter (void ); +void irq_enter (void ); +/* +* Like irq_enter(), but RCU is already watching. +*/ +void irq_enter_rcu (void );/* * Exit irq context without processing softirqs:
@@ -55,7 +59,12 @@ extern void irq_enter(void);/* * Exit irq context and process softirqs if needed: */ -extern void irq_exit (void ); +void irq_exit (void ); + +/* +* Like irq_exit(), but return with RCU watching. +*/ +void irq_exit_rcu (void );#ifndef arch_nmi_enter#define arch_nmi_enter () do { } while (0)
kernel/softirq.c +27 -8
@@ -339,12 +339,11 @@ asmlinkage __visible void do_softirq(void)local_irq_restore (flags); } -/* -* Enter an interrupt context. +/** +* irq_enter_rcu - Enter an interrupt context with RCU watching */ -void irq_enter (void ) +void irq_enter_rcu (void ) { -rcu_irq_enter ();if (is_idle_task (current) && !in_interrupt ()) {/* * Prevent raise_softirq from needlessly waking up ksoftirqd
@@ -354,10 +353,18 @@ void irq_enter(void)tick_irq_enter ();_local_bh_enable (); } -__irq_enter (); } +/** +* irq_enter - Enter an interrupt context including RCU update +*/ +void irq_enter (void ) +{ +rcu_irq_enter (); +irq_enter_rcu (); +} +static inline void invoke_softirq (void ) {if (ksoftirqd_running (local_softirq_pending ()))
@@ -397,10 +404,12 @@ static inline void tick_irq_exit(void)#endif } -/* -* Exit an interrupt context. Process softirqs if needed and possible: +/** +* irq_exit_rcu() - Exit an interrupt context without updating RCU +* +* Also processes softirqs if needed and possible. */ -void irq_exit (void ) +void irq_exit_rcu (void ) {#ifndef __ARCH_IRQ_EXIT_IRQS_DISABLEDlocal_irq_disable ();
@@ -413,6 +422,16 @@ void irq_exit(void)invoke_softirq ();tick_irq_exit (); +} + +/** +* irq_exit - Exit an interrupt context, update RCU and lockdep +* +* Also processes softirqs if needed and possible. +*/ +void irq_exit (void ) +{ +irq_exit_rcu ();rcu_irq_exit ();/* must be last! */ lockdep_hardirq_exit ();
PATCH v9 9/39 genirq: Provide __irq_enter/exit_raw()
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Like __irq_enter/exit() but without time accounting. To be used for "empty" system vectors like the scheduler IPI to avoid the overhead.
--- include/linux/hardirq.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
include/linux/hardirq.h +20 -0
@@ -50,6 +50,17 @@ static __always_inline void rcu_irq_ente }while (0 )/* +* Like __irq_enter() without time accounting for fast +* interrupts, e.g. reschedule IPI where time accounting +* is more expensive than the actual interrupt. + */ +#define __irq_enter_raw () \ + do { \ + preempt_count_add(HARDIRQ_OFFSET); \ + lockdep_hardirq_enter(); \ + } while (0) + +/* * Enter irq context (on NO_HZ, update jiffies): */ void irq_enter (void );
@@ -68,6 +79,15 @@ void irq_enter_rcu(void);preempt_count_sub (HARDIRQ_OFFSET ); \ }while (0 ) +/* +* Like __irq_exit() without time accounting + */ +#define __irq_exit_raw () \ + do { \ + lockdep_hardirq_exit(); \ + preempt_count_sub(HARDIRQ_OFFSET); \ + } while (0) +/* * Exit irq context and process softirqs if needed: */
[tip: x86/entry] genirq: Provide __irq_enter/exit_raw()
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 355e1262d6038b37f0dbf5f3cfe64cc85d471b7b Gitweb: https://git.kernel.org/tip/355e1262d6038b37f0dbf5f3cfe64cc85d471b7b Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:22 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
genirq: Provide __irq_enter/exit_raw()
Like __irq_enter/exit() but without time accounting. To be used for "empty" system vectors like the scheduler IPI to avoid the overhead.
--- include/linux/hardirq.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
include/linux/hardirq.h +20 -0
@@ -38,6 +38,17 @@ static __always_inline void rcu_irq_enter_check_tick(void) }while (0 )/* +* Like __irq_enter() without time accounting for fast +* interrupts, e.g. reschedule IPI where time accounting +* is more expensive than the actual interrupt. + */ +#define __irq_enter_raw () \ + do { \ + preempt_count_add(HARDIRQ_OFFSET); \ + lockdep_hardirq_enter(); \ + } while (0) + +/* * Enter irq context (on NO_HZ, update jiffies): */ void irq_enter (void );
@@ -57,6 +68,15 @@ void irq_enter_rcu(void); }while (0 )/* +* Like __irq_exit() without time accounting + */ +#define __irq_exit_raw () \ + do { \ + lockdep_hardirq_exit(); \ + preempt_count_sub(HARDIRQ_OFFSET); \ + } while (0) + +/* * Exit irq context and process softirqs if needed: */ void irq_exit (void );
PATCH v9 10/39 x86/entry: Provide helpers for execute on irqstack
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.
---
V9: Moved the conditions into an inline to avoid code duplication
--- arch/x86/entry/entry_64.S | 39 ++++++++++++++++++++++++++++ arch/x86/include/asm/irq_stack.h | 53 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+)
arch/x86/entry/entry_64.S +39 -0
@@ -1106,6 +1106,45 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs) SYM_CODE_END(.Lbad_gs) .previous +/* + * rdi: New stack pointer points to the top word of the stack + * rsi: Function pointer + * rdx: Function argument (can be NULL if none) + */ +SYM_FUNC_START(asm_call_on_stack) + /* + * Save the frame pointer unconditionally. This allows the ORC + * unwinder to handle the stack switch. + */ + pushq %rbp + mov %rsp, %rbp + + /* + * The unwinder relies on the word at the top of the new stack + * page linking back to the previous RSP. + */ + mov %rsp, (%rdi) + mov %rdi, %rsp + /* Move the argument to the right place */ + mov %rdx, %rdi + +1: + .pushsection .discard.instr_begin + .long 1b - . + .popsection + + CALL_NOSPEC rsi + +2: + .pushsection .discard.instr_end + .long 2b - . + .popsection + + /* Restore the previous stack pointer from RBP. */ + leaveq + ret +SYM_FUNC_END(asm_call_on_stack) + /* Call softirq on interrupt stack. Interrupts are off. */ .pushsection .text, "ax" SYM_FUNC_START(do_softirq_own_stack)
arch/x86/include/asm/irq_stack.h new file +53 -0
53 lines not on this page · raw message
[tip: x86/entry] x86/entry: Provide helpers for executing on the irqstack
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 0aa4dbb2808991f53396df8d2deb390d4f880abb Gitweb: https://git.kernel.org/tip/0aa4dbb2808991f53396df8d2deb390d4f880abb Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:23 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
x86/entry: Provide helpers for executing on the irqstack
Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows the moving of the stack switching to C code, which simplifies the entry/exit handling further, because it allows the switching of stacks after handling the entry and on exit before handling RCU, returning to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable, as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64-bit only for now, as 32-bit has a variant of that already. Once this is cleaned up, the two implementations might be consolidated as an additional cleanup on top.
--- arch/x86/entry/entry_64.S | 39 +++++++++++++++++++++++- arch/x86/include/asm/irq_stack.h | 53 +++++++++++++++++++++++++++++++- 2 files changed, 92 insertions(+) create mode 100644 arch/x86/include/asm/irq_stack.h
arch/x86/entry/entry_64.S +39 -0
@@ -1106,6 +1106,45 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs) SYM_CODE_END(.Lbad_gs) .previous +/* + * rdi: New stack pointer points to the top word of the stack + * rsi: Function pointer + * rdx: Function argument (can be NULL if none) + */ +SYM_FUNC_START(asm_call_on_stack) + /* + * Save the frame pointer unconditionally. This allows the ORC + * unwinder to handle the stack switch. + */ + pushq %rbp + mov %rsp, %rbp + + /* + * The unwinder relies on the word at the top of the new stack + * page linking back to the previous RSP. + */ + mov %rsp, (%rdi) + mov %rdi, %rsp + /* Move the argument to the right place */ + mov %rdx, %rdi + +1: + .pushsection .discard.instr_begin + .long 1b - . + .popsection + + CALL_NOSPEC rsi + +2: + .pushsection .discard.instr_end + .long 2b - . + .popsection + + /* Restore the previous stack pointer from RBP. */ + leaveq + ret +SYM_FUNC_END(asm_call_on_stack) + /* Call softirq on interrupt stack. Interrupts are off. */ .pushsection .text, "ax" SYM_FUNC_START(do_softirq_own_stack)
arch/x86/include/asm/irq_stack.h new file +53 -0
53 lines not on this page · raw message
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
In reply to Thomas GleixnerQuoted 23 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Quoted 8 lines
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.
Signed-off-by Thomas Gleixner <[email protected]>
Reverted this commit and the rest of series (with trivial fixup) as well as the two dependencies [1],
8449e768dcb8 ("x86/entry: Remove debug IDT frobbing") 029149180d1d ("x86/entry: Rename trace_hardirqs_off_prepare()")
fixed the warning under some memory pressure on AMD NUMA servers.
[ 9371.959858] asm_call_on_stack+0x12/0x20
asm_call_on_stack at arch/x86/entry/entry_64.S:710
The .config (if ever matters),
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config
[ 9371.260161] ------------[ cut here ]------------ [ 9371.267143] Stack depot reached limit capacity [ 9371.267193] WARNING: CPU: 19 PID: 1181 at lib/stackdepot.c:115 stack_depot_save+0x3d9/0x57d [ 9371.281470] Modules linked in: brd vfat fat ext4 crc16 mbcache jbd2 loop kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core acpi_cpufreq ip_tables x_table s xfs sd_mod bnxt_en smartpqi scsi_transport_sas tg3 i40e libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_del_mod] [ 9371.310176] CPU: 19 PID: 1181 Comm: systemd-journal Tainted: G O 5.7.0-next-20200604+ #1 [ 9371.320700] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 03/09/2018 [ 9371.329987] RIP: 0010:stack_depot_save+0x3d9/0x57d [ 9371.335513] Code: 1d 9b bc 68 01 80 fb 01 0f 87 c0 01 00 00 80 e3 01 75 1f 4c 89 45 c0 c6 05 82 bc 68 01 01 48 c7 c7 e0 85 63 9f e8 9d 74 9d ff <0f> 0b 90 90 4c 8 b 45 c0 48 c7 c7 80 1a d0 9f 4c 89 c6 e8 b0 2b 46 [ 9371.355426] RSP: 0018:ffffc90007260490 EFLAGS: 00010082 [ 9371.361387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff9ed3207f [ 9371.369544] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: 0000000000000000 [ 9371.377428] RBP: ffffc900072604f8 R08: fffffbfff3f37539 R09: fffffbfff3f37539 [ 9371.385310] R10: ffffffff9f9ba9c3 R11: fffffbfff3f37538 R12: ffffc90007260508 [ 9371.393521] R13: 0000000000000036 R14: 0000000000000000 R15: 000000000009fb52 [ 9371.401403] FS: 00007fc9849f2980(0000) GS:ffff88942fb80000(0000) knlGS:0000000000000000 [ 9371.410244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9371.417007] CR2: 00007f20d02c3000 CR3: 0000000440b9c000 CR4: 00000000003406e0 [ 9371.424889] Call Trace: [ 9371.428054] <IRQ> [ 9371.436315] save_stack+0x3f/0x50 [ 9371.734034] kasan_slab_free+0xe/0x10 [ 9371.738798] slab_free_freelist_hook+0x5d/0x1c0 [ 9371.748886] kmem_cache_free+0x10c/0x390 [ 9371.758450] mempool_free_slab+0x17/0x20 [ 9371.763522] mempool_free+0x65/0x170 [ 9371.767825] bio_free+0x14c/0x210 [ 9371.771864] bio_put+0x59/0x70 [ 9371.775644] end_swap_bio_write+0x199/0x250 [ 9371.780556] bio_endio+0x22c/0x4e0 [ 9371.784709] dec_pending+0x1bf/0x3e0 [dm_mod] [ 9371.790207] clone_endio+0x129/0x3d0 [dm_mod] [ 9371.800746] bio_endio+0x22c/0x4e0 [ 9371.809262] blk_update_request+0x3bb/0x980 [ 9371.814605] scsi_end_request+0x53/0x420 [ 9371.824002] scsi_io_completion+0x10a/0x830 [ 9371.844445] scsi_finish_command+0x1b9/0x250 [ 9371.849445] scsi_softirq_done+0x1ab/0x1f0 [ 9371.854272] blk_mq_force_complete_rq+0x217/0x250 [ 9371.859708] blk_mq_complete_request+0xe/0x20 [ 9371.865171] scsi_mq_done+0xc1/0x220 [ 9371.869479] pqi_aio_io_complete+0x83/0x2c0 [smartpqi] [ 9371.881764] pqi_irq_handler+0x1fc/0x13f0 [smartpqi] [ 9371.914115] __handle_irq_event_percpu+0x81/0x550 [ 9371.924289] handle_irq_event_percpu+0x70/0x100 [ 9371.945559] handle_irq_event+0x5a/0x8b [ 9371.950121] handle_edge_irq+0x10c/0x370 [ 9371.950121] handle_edge_irq+0x10c/0x370 [ 9371.959858] asm_call_on_stack+0x12/0x20 asm_call_on_stack at arch/x86/entry/entry_64.S:710 [ 9371.964899] </IRQ> [ 9371.967716] common_interrupt+0x185/0x2a0 [ 9371.972455] asm_common_interrupt+0x1e/0x40 [ 9371.977368] RIP: 0010:__asan_load4+0x8/0xa0 [ 9371.982281] Code: 00 e8 5c f4 ff ff 5d c3 40 38 f0 0f 9e c0 84 c0 75 e5 5d c3 48 c1 e8 03 80 3c 10 00 75 ed 5d c3 66 90 55 48 89 e5 48 8b 4d 08 <48> 83 ff fb 77 6c eb 3a 0f 1f 00 48 b8 00 00 00 00 00 00 00 ff 48 [ 9372.002246] RSP: 0018:ffffc9000b12f0e8 EFLAGS: 00000202 [ 9372.008207] RAX: 0000000000000000 RBX: ffffc9000b12f150 RCX: ffffffff9ed32487 [ 9372.016489] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffffffff9ff92a94 [ 9372.024370] RBP: ffffc9000b12f0e8 R08: fffffbfff3ff1d4d R09: fffffbfff3ff1d4d [ 9372.032252] R10: ffffffff9ff8ea67 R11: fffffbfff3ff1d4c R12: 0000000000000000 [ 9372.040490] R13: ffffc9000b12f358 R14: 000000000000000c R15: 0000000000000005 [ 9372.053899] debug_lockdep_rcu_enabled+0x27/0x60 [ 9372.059248] rcu_read_lock_held_common+0x12/0x60 [ 9372.064989] rcu_read_lock_sched_held+0x60/0xe0 [ 9372.080338] shrink_active_list+0xbfd/0xc30 [ 9372.120481] shrink_lruvec+0xbf1/0x11b0 [ 9372.150410] shrink_node+0x344/0xd10 [ 9372.154719] do_try_to_free_pages+0x263/0xa00 [ 9372.169860] try_to_free_pages+0x239/0x570 [ 9372.179356] __alloc_pages_slowpath.constprop.59+0x5dd/0x1880 [ 9372.215762] __alloc_pages_nodemask+0x562/0x670 [ 9372.232686] alloc_pages_current+0x9c/0x110 [ 9372.237599] alloc_slab_page+0x355/0x530 [ 9372.242755] allocate_slab+0x485/0x5a0 [ 9372.247232] new_slab+0x46/0x70 [ 9372.251095] ___slab_alloc+0x35f/0x810 [ 9372.274485] __slab_alloc+0x43/0x70 [ 9372.287650] kmem_cache_alloc+0x257/0x3d0 [ 9372.298182] prepare_creds+0x26/0x130 [ 9372.302571] do_faccessat+0x255/0x3e0 [ 9372.321591] __x64_sys_access+0x38/0x40 [ 9372.326154] do_syscall_64+0x64/0x340 [ 9372.330542] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 9372.336329] RIP: 0033:0x7fc983a21bfb [ 9372.341050] Code: Bad RIP value. [ 9372.345000] RSP: 002b:00007fffd543cde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000015 [ 9372.353319] RAX: ffffffffffffffda RBX: 00007fffd543fa40 RCX: 00007fc983a21bfb [ 9372.361199] RDX: 00007fc983cf1c00 RSI: 0000000000000000 RDI: 00005573ed458090 [ 9372.369512] RBP: 00007fffd543cf30 R08: 00005573ed44ab99 R09: 0000000000000007 [ 9372.377393] R10: 0000000000000041 R11: 0000000000000246 R12: 0000000000000000 [ 9372.385275] R13: 0000000000000000 R14: 00007fffd543cea0 R15: 00005573eecd2990 [ 9372.393608] irq event stamp: 27274496 [ 9372.397999] hardirqs last enabled at (27274495): [<ffffffff9e635b8e>] free_unref_page_list+0x2ee/0x400 [ 9372.408152] hardirqs last disabled at (27274496): [<ffffffff9ed2c22b>] idtentry_enter_cond_rcu+0x1b/0x50 [ 9372.418695] softirqs last enabled at (27272816): [<ffffffff9f000478>] __do_softirq+0x478/0x784 [ 9372.428154] softirqs last disabled at (27272807): [<ffffffff9e2d0b41>] irq_exit_rcu+0xd1/0xe0 [ 9372.437435] ---[ end trace d2ebac1fad6e452e ]---
[1]
git revert --no-edit 355e1262d603..5a7462b1f9c1
Quoted 112 lines
---
V9: Moved the conditions into an inline to avoid code duplication--- arch/x86/entry/entry_64.S | 39 ++++++++++++++++++++++++++++ arch/x86/include/asm/irq_stack.h | 53 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+)arch/x86/entry/entry_64.S +39
-0@@ -1106,6 +1106,45 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs) SYM_CODE_END(.Lbad_gs) .previous +/* + * rdi: New stack pointer points to the top word of the stack + * rsi: Function pointer + * rdx: Function argument (can be NULL if none) + */ +SYM_FUNC_START(asm_call_on_stack) + /* + * Save the frame pointer unconditionally. This allows the ORC + * unwinder to handle the stack switch. + */ + pushq %rbp + mov %rsp, %rbp + + /* + * The unwinder relies on the word at the top of the new stack + * page linking back to the previous RSP. + */ + mov %rsp, (%rdi) + mov %rdi, %rsp + /* Move the argument to the right place */ + mov %rdx, %rdi + +1: + .pushsection .discard.instr_begin + .long 1b - . + .popsection + + CALL_NOSPEC rsi + +2: + .pushsection .discard.instr_end + .long 2b - . + .popsection + + /* Restore the previous stack pointer from RBP. */ + leaveq + ret +SYM_FUNC_END(asm_call_on_stack) + /* Call softirq on interrupt stack. Interrupts are off. */ .pushsection .text, "ax" SYM_FUNC_START(do_softirq_own_stack)arch/x86/include/asm/irq_stack.h new file +53
-0@@ -0,0 +1,53 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_IRQ_STACK_H +#define _ASM_X86_IRQ_STACK_H + +#include <linux/ptrace.h> + +#include <asm/processor.h> + +#ifdef CONFIG_X86_64 +static __always_inline bool irqstack_active(void) +{ +return __this_cpu_read (irq_count) !=-1 ; +} + +void asm_call_on_stack (void *sp,void *func,void *arg); + +static __always_inline void __run_on_irqstack(void *func, void *arg) +{ +void *tos =__this_cpu_read (hardirq_stack_ptr); + +__this_cpu_add (irq_count,1 ); +asm_call_on_stack (tos -8 , func, arg); +__this_cpu_sub (irq_count,1 ); +} + +#else /* CONFIG_X86_64 */ +static inline bool irqstack_active (void ) {return false; } +static inline void __run_on_irqstack (void *func,void *arg) { } +#endif /* !CONFIG_X86_64 */ + +static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs) +{ +if (IS_ENABLED (CONFIG_X86_32 )) +return false; +if (!regs) +return !irqstack_active (); +return !user_mode (regs) && !irqstack_active (); +} + +static __always_inline void run_on_irqstack_cond(void *func, void *arg, +struct pt_regs *regs) +{ +void (*__func)(void *arg) = func; + +lockdep_assert_irqs_disabled (); + +if (irq_needs_irq_stack (regs)) +__run_on_irqstack (__func, arg); +else +__func (arg); +} + +#endif
Peter Zijlstra
maintainerRe: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 33 lines · Qian Cai
Quoted 23 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Quoted 8 lines
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.
Signed-off-by Thomas Gleixner <[email protected]>
Quoted 7 lines
Reverted this commit and the rest of series (with trivial fixup) as well as the two dependencies [1],
8449e768dcb8("x86/entry: Remove debug IDT frobbing")029149180d1d("x86/entry: Renametrace_hardirqs_off_prepare()")fixed the warning under some memory pressure on AMD NUMA servers.
The warning ?
Quoted 2 lines
[ 9371.959858] asm_call_on_stack+0x12/0x20
asm_call_on_stackatarch/x86/entry/entry_64.S:710
^^ what's that?
Quoted 6 lines
The .config (if ever matters),
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config[ 9371.260161] ------------[ cut here ]------------ [ 9371.267143] Stack depot reached limit capacity [ 9371.267193] WARNING: CPU: 19 PID: 1181 at lib/stackdepot.c:115 stack_depot_save+0x3d9/0x57d
Is this _the_ warning?
Maybe it really is running out of storage, I don't think this patch is to blame. The unwind here looks good, so why would the stack-depot unwind be any different?
Quoted 95 lines
[ 9371.281470] Modules linked in: brd vfat fat ext4 crc16 mbcache jbd2 loop kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core acpi_cpufreq ip_tables x_table s xfs sd_mod bnxt_en smartpqi scsi_transport_sas tg3 i40e libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_del_mod] [ 9371.310176] CPU: 19 PID: 1181 Comm: systemd-journal Tainted: G O 5.7.0-next-20200604+ #1 [ 9371.320700] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 03/09/2018 [ 9371.329987] RIP: 0010:stack_depot_save+0x3d9/0x57d [ 9371.335513] Code: 1d 9b bc 68 01 80 fb 01 0f 87 c0 01 00 00 80 e3 01 75 1f 4c 89 45 c0 c6 05 82 bc 68 01 01 48 c7 c7 e0 85 63 9f e8 9d 74 9d ff <0f> 0b 90 90 4c 8 b 45 c0 48 c7 c7 80 1a d0 9f 4c 89 c6 e8 b0 2b 46 [ 9371.355426] RSP: 0018:ffffc90007260490 EFLAGS: 00010082 [ 9371.361387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff9ed3207f [ 9371.369544] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: 0000000000000000 [ 9371.377428] RBP: ffffc900072604f8 R08: fffffbfff3f37539 R09: fffffbfff3f37539 [ 9371.385310] R10: ffffffff9f9ba9c3 R11: fffffbfff3f37538 R12: ffffc90007260508 [ 9371.393521] R13: 0000000000000036 R14: 0000000000000000 R15: 000000000009fb52 [ 9371.401403] FS: 00007fc9849f2980(0000) GS:ffff88942fb80000(0000) knlGS:0000000000000000 [ 9371.410244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9371.417007] CR2: 00007f20d02c3000 CR3: 0000000440b9c000 CR4: 00000000003406e0 [ 9371.424889] Call Trace: [ 9371.428054] <IRQ> [ 9371.436315] save_stack+0x3f/0x50 [ 9371.734034] kasan_slab_free+0xe/0x10 [ 9371.738798] slab_free_freelist_hook+0x5d/0x1c0 [ 9371.748886] kmem_cache_free+0x10c/0x390 [ 9371.758450] mempool_free_slab+0x17/0x20 [ 9371.763522] mempool_free+0x65/0x170 [ 9371.767825] bio_free+0x14c/0x210 [ 9371.771864] bio_put+0x59/0x70 [ 9371.775644] end_swap_bio_write+0x199/0x250 [ 9371.780556] bio_endio+0x22c/0x4e0 [ 9371.784709] dec_pending+0x1bf/0x3e0 [dm_mod] [ 9371.790207] clone_endio+0x129/0x3d0 [dm_mod] [ 9371.800746] bio_endio+0x22c/0x4e0 [ 9371.809262] blk_update_request+0x3bb/0x980 [ 9371.814605] scsi_end_request+0x53/0x420 [ 9371.824002] scsi_io_completion+0x10a/0x830 [ 9371.844445] scsi_finish_command+0x1b9/0x250 [ 9371.849445] scsi_softirq_done+0x1ab/0x1f0 [ 9371.854272] blk_mq_force_complete_rq+0x217/0x250 [ 9371.859708] blk_mq_complete_request+0xe/0x20 [ 9371.865171] scsi_mq_done+0xc1/0x220 [ 9371.869479] pqi_aio_io_complete+0x83/0x2c0 [smartpqi] [ 9371.881764] pqi_irq_handler+0x1fc/0x13f0 [smartpqi] [ 9371.914115] __handle_irq_event_percpu+0x81/0x550 [ 9371.924289] handle_irq_event_percpu+0x70/0x100 [ 9371.945559] handle_irq_event+0x5a/0x8b [ 9371.950121] handle_edge_irq+0x10c/0x370 [ 9371.950121] handle_edge_irq+0x10c/0x370 [ 9371.959858] asm_call_on_stack+0x12/0x20 asm_call_on_stack at arch/x86/entry/entry_64.S:710 [ 9371.964899] </IRQ> [ 9371.967716] common_interrupt+0x185/0x2a0 [ 9371.972455] asm_common_interrupt+0x1e/0x40 [ 9371.977368] RIP: 0010:__asan_load4+0x8/0xa0 [ 9371.982281] Code: 00 e8 5c f4 ff ff 5d c3 40 38 f0 0f 9e c0 84 c0 75 e5 5d c3 48 c1 e8 03 80 3c 10 00 75 ed 5d c3 66 90 55 48 89 e5 48 8b 4d 08 <48> 83 ff fb 77 6c eb 3a 0f 1f 00 48 b8 00 00 00 00 00 00 00 ff 48 [ 9372.002246] RSP: 0018:ffffc9000b12f0e8 EFLAGS: 00000202 [ 9372.008207] RAX: 0000000000000000 RBX: ffffc9000b12f150 RCX: ffffffff9ed32487 [ 9372.016489] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffffffff9ff92a94 [ 9372.024370] RBP: ffffc9000b12f0e8 R08: fffffbfff3ff1d4d R09: fffffbfff3ff1d4d [ 9372.032252] R10: ffffffff9ff8ea67 R11: fffffbfff3ff1d4c R12: 0000000000000000 [ 9372.040490] R13: ffffc9000b12f358 R14: 000000000000000c R15: 0000000000000005 [ 9372.053899] debug_lockdep_rcu_enabled+0x27/0x60 [ 9372.059248] rcu_read_lock_held_common+0x12/0x60 [ 9372.064989] rcu_read_lock_sched_held+0x60/0xe0 [ 9372.080338] shrink_active_list+0xbfd/0xc30 [ 9372.120481] shrink_lruvec+0xbf1/0x11b0 [ 9372.150410] shrink_node+0x344/0xd10 [ 9372.154719] do_try_to_free_pages+0x263/0xa00 [ 9372.169860] try_to_free_pages+0x239/0x570 [ 9372.179356] __alloc_pages_slowpath.constprop.59+0x5dd/0x1880 [ 9372.215762] __alloc_pages_nodemask+0x562/0x670 [ 9372.232686] alloc_pages_current+0x9c/0x110 [ 9372.237599] alloc_slab_page+0x355/0x530 [ 9372.242755] allocate_slab+0x485/0x5a0 [ 9372.247232] new_slab+0x46/0x70 [ 9372.251095] ___slab_alloc+0x35f/0x810 [ 9372.274485] __slab_alloc+0x43/0x70 [ 9372.287650] kmem_cache_alloc+0x257/0x3d0 [ 9372.298182] prepare_creds+0x26/0x130 [ 9372.302571] do_faccessat+0x255/0x3e0 [ 9372.321591] __x64_sys_access+0x38/0x40 [ 9372.326154] do_syscall_64+0x64/0x340 [ 9372.330542] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 9372.336329] RIP: 0033:0x7fc983a21bfb [ 9372.341050] Code: Bad RIP value. [ 9372.345000] RSP: 002b:00007fffd543cde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000015 [ 9372.353319] RAX: ffffffffffffffda RBX: 00007fffd543fa40 RCX: 00007fc983a21bfb [ 9372.361199] RDX: 00007fc983cf1c00 RSI: 0000000000000000 RDI: 00005573ed458090 [ 9372.369512] RBP: 00007fffd543cf30 R08: 00005573ed44ab99 R09: 0000000000000007 [ 9372.377393] R10: 0000000000000041 R11: 0000000000000246 R12: 0000000000000000 [ 9372.385275] R13: 0000000000000000 R14: 00007fffd543cea0 R15: 00005573eecd2990 [ 9372.393608] irq event stamp: 27274496 [ 9372.397999] hardirqs last enabled at (27274495): [<ffffffff9e635b8e>] free_unref_page_list+0x2ee/0x400 [ 9372.408152] hardirqs last disabled at (27274496): [<ffffffff9ed2c22b>] idtentry_enter_cond_rcu+0x1b/0x50 [ 9372.418695] softirqs last enabled at (27272816): [<ffffffff9f000478>] __do_softirq+0x478/0x784 [ 9372.428154] softirqs last disabled at (27272807): [<ffffffff9e2d0b41>] irq_exit_rcu+0xd1/0xe0 [ 9372.437435] ---[ end trace d2ebac1fad6e452e ]---
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 37 lines · Peter Zijlstra
Quoted 36 lines · Qian Cai
Quoted 23 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Quoted 8 lines
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.
Signed-off-by Thomas Gleixner <[email protected]>Reverted this commit and the rest of series (with trivial fixup) as well as the two dependencies [1],
Quoted 6 lines
Quoted 4 lines
8449e768dcb8("x86/entry: Remove debug IDT frobbing")029149180d1d("x86/entry: Renametrace_hardirqs_off_prepare()")fixed the warning under some memory pressure on AMD NUMA servers.
The warning ?
Obviously, the warning below.
Quoted 3 lines
Quoted 3 lines
[ 9371.959858] asm_call_on_stack+0x12/0x20
asm_call_on_stackatarch/x86/entry/entry_64.S:710
This is one piece of call from the warning call traces that introduced by the patch which leads me to revert the commit in the first place. It may or may not be the real culprit, but just wanted to highlight it in case.
Quoted 10 lines
^^ what's that?
Quoted 6 lines
The .config (if ever matters),
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config[ 9371.260161] ------------[ cut here ]------------ [ 9371.267143] Stack depot reached limit capacity [ 9371.267193] WARNING: CPU: 19 PID: 1181 at lib/stackdepot.c:115 stack_depot_save+0x3d9/0x57d
Quoted 5 lines
Is this _the_ warning?
Maybe it really is running out of storage, I don't think this patch is to blame. The unwind here looks good, so why would the stack-depot unwind be any different?
Don't know. It is always reproducible with the same warning and call traces on multiple similar AMD servers. Can this patch (or series) increase the storage so much or some kind of recursion? Otherwise, never saw this problem before the series was introduced. It is always running almost the same tests for years on those machines.
Quoted 96 lines
Quoted 96 lines
[ 9371.281470] Modules linked in: brd vfat fat ext4 crc16 mbcache jbd2 loop kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core acpi_cpufreq ip_tables x_table s xfs sd_mod bnxt_en smartpqi scsi_transport_sas tg3 i40e libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_del_mod] [ 9371.310176] CPU: 19 PID: 1181 Comm: systemd-journal Tainted: G O 5.7.0-next-20200604+ #1 [ 9371.320700] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 03/09/2018 [ 9371.329987] RIP: 0010:stack_depot_save+0x3d9/0x57d [ 9371.335513] Code: 1d 9b bc 68 01 80 fb 01 0f 87 c0 01 00 00 80 e3 01 75 1f 4c 89 45 c0 c6 05 82 bc 68 01 01 48 c7 c7 e0 85 63 9f e8 9d 74 9d ff <0f> 0b 90 90 4c 8 b 45 c0 48 c7 c7 80 1a d0 9f 4c 89 c6 e8 b0 2b 46 [ 9371.355426] RSP: 0018:ffffc90007260490 EFLAGS: 00010082 [ 9371.361387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff9ed3207f [ 9371.369544] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: 0000000000000000 [ 9371.377428] RBP: ffffc900072604f8 R08: fffffbfff3f37539 R09: fffffbfff3f37539 [ 9371.385310] R10: ffffffff9f9ba9c3 R11: fffffbfff3f37538 R12: ffffc90007260508 [ 9371.393521] R13: 0000000000000036 R14: 0000000000000000 R15: 000000000009fb52 [ 9371.401403] FS: 00007fc9849f2980(0000) GS:ffff88942fb80000(0000) knlGS:0000000000000000 [ 9371.410244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9371.417007] CR2: 00007f20d02c3000 CR3: 0000000440b9c000 CR4: 00000000003406e0 [ 9371.424889] Call Trace: [ 9371.428054] <IRQ> [ 9371.436315] save_stack+0x3f/0x50 [ 9371.734034] kasan_slab_free+0xe/0x10 [ 9371.738798] slab_free_freelist_hook+0x5d/0x1c0 [ 9371.748886] kmem_cache_free+0x10c/0x390 [ 9371.758450] mempool_free_slab+0x17/0x20 [ 9371.763522] mempool_free+0x65/0x170 [ 9371.767825] bio_free+0x14c/0x210 [ 9371.771864] bio_put+0x59/0x70 [ 9371.775644] end_swap_bio_write+0x199/0x250 [ 9371.780556] bio_endio+0x22c/0x4e0 [ 9371.784709] dec_pending+0x1bf/0x3e0 [dm_mod] [ 9371.790207] clone_endio+0x129/0x3d0 [dm_mod] [ 9371.800746] bio_endio+0x22c/0x4e0 [ 9371.809262] blk_update_request+0x3bb/0x980 [ 9371.814605] scsi_end_request+0x53/0x420 [ 9371.824002] scsi_io_completion+0x10a/0x830 [ 9371.844445] scsi_finish_command+0x1b9/0x250 [ 9371.849445] scsi_softirq_done+0x1ab/0x1f0 [ 9371.854272] blk_mq_force_complete_rq+0x217/0x250 [ 9371.859708] blk_mq_complete_request+0xe/0x20 [ 9371.865171] scsi_mq_done+0xc1/0x220 [ 9371.869479] pqi_aio_io_complete+0x83/0x2c0 [smartpqi] [ 9371.881764] pqi_irq_handler+0x1fc/0x13f0 [smartpqi] [ 9371.914115] __handle_irq_event_percpu+0x81/0x550 [ 9371.924289] handle_irq_event_percpu+0x70/0x100 [ 9371.945559] handle_irq_event+0x5a/0x8b [ 9371.950121] handle_edge_irq+0x10c/0x370 [ 9371.950121] handle_edge_irq+0x10c/0x370 [ 9371.959858] asm_call_on_stack+0x12/0x20 asm_call_on_stack at arch/x86/entry/entry_64.S:710 [ 9371.964899] </IRQ> [ 9371.967716] common_interrupt+0x185/0x2a0 [ 9371.972455] asm_common_interrupt+0x1e/0x40 [ 9371.977368] RIP: 0010:__asan_load4+0x8/0xa0 [ 9371.982281] Code: 00 e8 5c f4 ff ff 5d c3 40 38 f0 0f 9e c0 84 c0 75 e5 5d c3 48 c1 e8 03 80 3c 10 00 75 ed 5d c3 66 90 55 48 89 e5 48 8b 4d 08 <48> 83 ff fb 77 6c eb 3a 0f 1f 00 48 b8 00 00 00 00 00 00 00 ff 48 [ 9372.002246] RSP: 0018:ffffc9000b12f0e8 EFLAGS: 00000202 [ 9372.008207] RAX: 0000000000000000 RBX: ffffc9000b12f150 RCX: ffffffff9ed32487 [ 9372.016489] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffffffff9ff92a94 [ 9372.024370] RBP: ffffc9000b12f0e8 R08: fffffbfff3ff1d4d R09: fffffbfff3ff1d4d [ 9372.032252] R10: ffffffff9ff8ea67 R11: fffffbfff3ff1d4c R12: 0000000000000000 [ 9372.040490] R13: ffffc9000b12f358 R14: 000000000000000c R15: 0000000000000005 [ 9372.053899] debug_lockdep_rcu_enabled+0x27/0x60 [ 9372.059248] rcu_read_lock_held_common+0x12/0x60 [ 9372.064989] rcu_read_lock_sched_held+0x60/0xe0 [ 9372.080338] shrink_active_list+0xbfd/0xc30 [ 9372.120481] shrink_lruvec+0xbf1/0x11b0 [ 9372.150410] shrink_node+0x344/0xd10 [ 9372.154719] do_try_to_free_pages+0x263/0xa00 [ 9372.169860] try_to_free_pages+0x239/0x570 [ 9372.179356] __alloc_pages_slowpath.constprop.59+0x5dd/0x1880 [ 9372.215762] __alloc_pages_nodemask+0x562/0x670 [ 9372.232686] alloc_pages_current+0x9c/0x110 [ 9372.237599] alloc_slab_page+0x355/0x530 [ 9372.242755] allocate_slab+0x485/0x5a0 [ 9372.247232] new_slab+0x46/0x70 [ 9372.251095] ___slab_alloc+0x35f/0x810 [ 9372.274485] __slab_alloc+0x43/0x70 [ 9372.287650] kmem_cache_alloc+0x257/0x3d0 [ 9372.298182] prepare_creds+0x26/0x130 [ 9372.302571] do_faccessat+0x255/0x3e0 [ 9372.321591] __x64_sys_access+0x38/0x40 [ 9372.326154] do_syscall_64+0x64/0x340 [ 9372.330542] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 9372.336329] RIP: 0033:0x7fc983a21bfb [ 9372.341050] Code: Bad RIP value. [ 9372.345000] RSP: 002b:00007fffd543cde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000015 [ 9372.353319] RAX: ffffffffffffffda RBX: 00007fffd543fa40 RCX: 00007fc983a21bfb [ 9372.361199] RDX: 00007fc983cf1c00 RSI: 0000000000000000 RDI: 00005573ed458090 [ 9372.369512] RBP: 00007fffd543cf30 R08: 00005573ed44ab99 R09: 0000000000000007 [ 9372.377393] R10: 0000000000000041 R11: 0000000000000246 R12: 0000000000000000 [ 9372.385275] R13: 0000000000000000 R14: 00007fffd543cea0 R15: 00005573eecd2990 [ 9372.393608] irq event stamp: 27274496 [ 9372.397999] hardirqs last enabled at (27274495): [<ffffffff9e635b8e>] free_unref_page_list+0x2ee/0x400 [ 9372.408152] hardirqs last disabled at (27274496): [<ffffffff9ed2c22b>] idtentry_enter_cond_rcu+0x1b/0x50 [ 9372.418695] softirqs last enabled at (27272816): [<ffffffff9f000478>] __do_softirq+0x478/0x784 [ 9372.428154] softirqs last disabled at (27272807): [<ffffffff9e2d0b41>] irq_exit_rcu+0xd1/0xe0 [ 9372.437435] ---[ end trace d2ebac1fad6e452e ]---
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 8 lines · Qian Cai
Quoted 2 lines · Peter Zijlstra
Quoted 2 lines
[ 9371.959858] asm_call_on_stack+0x12/0x20
asm_call_on_stackatarch/x86/entry/entry_64.S:710This is one piece of call from the warning call traces that introduced by the patch which leads me to revert the commit in the first place. It may or may not be the real culprit, but just wanted to highlight it in case.
Oh well. The warning is a storage check in the stack depot code, i.e. stack depot ran out of storage space.
Even if that commit causes stack traces to be larger that revert does not make any sense at all and handwaving about recursions does not help either. If that commit introduced a recursion then that would have worse effects than triggering this warning.
The difference between the interrupt stack switching introduced by this commit is that it generates another entry in the stack trace compared to the state before it, which obviously has an effect on the storage requirements in the stack depot.
Thanks,
tglx
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 21 lines · Thomas Gleixner
CC + AlexanderQuoted 8 lines · Qian Cai
Quoted 2 lines · Peter Zijlstra
Quoted 2 lines
[ 9371.959858] asm_call_on_stack+0x12/0x20
asm_call_on_stackatarch/x86/entry/entry_64.S:710This is one piece of call from the warning call traces that introduced by the patch which leads me to revert the commit in the first place. It may or may not be the real culprit, but just wanted to highlight it in case.
Oh well. The warning is a storage check in the stack depot code, i.e. stack depot ran out of storage space.
Even if that commit causes stack traces to be larger that revert does not make any sense at all and handwaving about recursions does not help either. If that commit introduced a recursion then that would have worse effects than triggering this warning.
Quoted 4 lines
The difference between the interrupt stack switching introduced by this commit is that it generates another entry in the stack trace compared to the state before it, which obviously has an effect on the storage requirements in the stack depot.
Sorry, not meant to propose the reverting as a final solution to the problem, but rather a data point. My problem is that I knew so little about this kernel area, so I was to throw out something where I saw something similiar while working in other areas over the years. Of course, my "random" tips could be wrong.
The effect is quite bad for our CI because it will disable lockdep in the middle of the tests which could miss regression bugs lockdep might find later. Anyway, I am not motivate enough to measure how much "an effect" is yet due to this patchset. I'll trim our .config and tests to hopefully not running out of the storage this early.
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
In reply to Peter ZijlstraQuoted 37 lines · Peter Zijlstra
Quoted 36 lines · Qian Cai
Quoted 23 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Quoted 8 lines
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.
Signed-off-by Thomas Gleixner <[email protected]>Reverted this commit and the rest of series (with trivial fixup) as well as the two dependencies [1],
Quoted 9 lines
Quoted 4 lines
8449e768dcb8("x86/entry: Remove debug IDT frobbing")029149180d1d("x86/entry: Renametrace_hardirqs_off_prepare()")fixed the warning under some memory pressure on AMD NUMA servers.
The warning ?
Quoted 2 lines
[ 9371.959858] asm_call_on_stack+0x12/0x20
asm_call_on_stackatarch/x86/entry/entry_64.S:710
Even after I trimmed the .config [1] to the barely minimal, a subset of LTP test still unable to finish on those AMD servers with page_owner=on.
[1]:
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config
It looks like because this new IRQ entry introduced by this patch,
</IRQ>
asm_call_on_stack at arch/x86/entry/entry_64.S:710 handle_edge_irq at kernel/irq/chip.c:832
which will running out of the stack depot limit due to nested loops below.
which has this loop,
do {
...
handle_irq_event(desc);
...
} while ((desc->istate & IRQS_PENDING) &&
!irqd_irq_disabled(&desc->irq_data));handle_irq_event at kernel/irq/handle.c:215 __handle_irq_event_percpu at kernel/irq/handle.c:156
Here has a nested loop,
for_each_action_of_desc(desc, action) {
...
res = action->handler(irq, action->dev_id);
...
}pqi_irq_handler at drivers/scsi/smartpqi/smartpqi_init.c:3462 pqi_process_io_intr at drivers/scsi/smartpqi/smartpqi_init.c:2992
Here has another nested loop,
while (1) {
...
io_request->io_complete_callback(io_request,
io_request->context);
...
}scsi_mq_done at drivers/scsi/scsi_lib.c:1602blk_mq_complete_request at block/blk-mq.c:677blk_mq_force_complete_rq at block/blk-mq.c:637scsi_io_completion at drivers/scsi/scsi_lib.c:934scsi_end_request at drivers/scsi/scsi_lib.c:584scsi_mq_uninit_cmd at drivers/scsi/scsi_lib.c:547scsi_free_sgtables at drivers/scsi/scsi_lib.c:537__sg_free_table at lib/scatterlist.c:225
Here has one more nested loop,
while (table->orig_nents) {
...
free_fn(sgl, alloc_size);
...
}free_fn() will call kmem_cache_free(). Since we have page_owner=on, it will call save_stack() to save the each free stack trace.
Quoted 106 lines
Quoted 4 lines
[ 9371.260161] ------------[ cut here ]------------ [ 9371.267143] Stack depot reached limit capacity [ 9371.267193] WARNING: CPU: 19 PID: 1181 at lib/stackdepot.c:115 stack_depot_save+0x3d9/0x57dIs this _the_ warning?
Maybe it really is running out of storage, I don't think this patch is to blame. The unwind here looks good, so why would the stack-depot unwind be any different?
Quoted 95 lines
[ 9371.281470] Modules linked in: brd vfat fat ext4 crc16 mbcache jbd2 loop kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core acpi_cpufreq ip_tables x_table s xfs sd_mod bnxt_en smartpqi scsi_transport_sas tg3 i40e libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_del_mod] [ 9371.310176] CPU: 19 PID: 1181 Comm: systemd-journal Tainted: G O 5.7.0-next-20200604+ #1 [ 9371.320700] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 03/09/2018 [ 9371.329987] RIP: 0010:stack_depot_save+0x3d9/0x57d [ 9371.335513] Code: 1d 9b bc 68 01 80 fb 01 0f 87 c0 01 00 00 80 e3 01 75 1f 4c 89 45 c0 c6 05 82 bc 68 01 01 48 c7 c7 e0 85 63 9f e8 9d 74 9d ff <0f> 0b 90 90 4c 8 b 45 c0 48 c7 c7 80 1a d0 9f 4c 89 c6 e8 b0 2b 46 [ 9371.355426] RSP: 0018:ffffc90007260490 EFLAGS: 00010082 [ 9371.361387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff9ed3207f [ 9371.369544] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: 0000000000000000 [ 9371.377428] RBP: ffffc900072604f8 R08: fffffbfff3f37539 R09: fffffbfff3f37539 [ 9371.385310] R10: ffffffff9f9ba9c3 R11: fffffbfff3f37538 R12: ffffc90007260508 [ 9371.393521] R13: 0000000000000036 R14: 0000000000000000 R15: 000000000009fb52 [ 9371.401403] FS: 00007fc9849f2980(0000) GS:ffff88942fb80000(0000) knlGS:0000000000000000 [ 9371.410244] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9371.417007] CR2: 00007f20d02c3000 CR3: 0000000440b9c000 CR4: 00000000003406e0 [ 9371.424889] Call Trace: [ 9371.428054] <IRQ> [ 9371.436315] save_stack+0x3f/0x50 [ 9371.734034] kasan_slab_free+0xe/0x10 [ 9371.738798] slab_free_freelist_hook+0x5d/0x1c0 [ 9371.748886] kmem_cache_free+0x10c/0x390 [ 9371.758450] mempool_free_slab+0x17/0x20 [ 9371.763522] mempool_free+0x65/0x170 [ 9371.767825] bio_free+0x14c/0x210 [ 9371.771864] bio_put+0x59/0x70 [ 9371.775644] end_swap_bio_write+0x199/0x250 [ 9371.780556] bio_endio+0x22c/0x4e0 [ 9371.784709] dec_pending+0x1bf/0x3e0 [dm_mod] [ 9371.790207] clone_endio+0x129/0x3d0 [dm_mod] [ 9371.800746] bio_endio+0x22c/0x4e0 [ 9371.809262] blk_update_request+0x3bb/0x980 [ 9371.814605] scsi_end_request+0x53/0x420 [ 9371.824002] scsi_io_completion+0x10a/0x830 [ 9371.844445] scsi_finish_command+0x1b9/0x250 [ 9371.849445] scsi_softirq_done+0x1ab/0x1f0 [ 9371.854272] blk_mq_force_complete_rq+0x217/0x250 [ 9371.859708] blk_mq_complete_request+0xe/0x20 [ 9371.865171] scsi_mq_done+0xc1/0x220 [ 9371.869479] pqi_aio_io_complete+0x83/0x2c0 [smartpqi] [ 9371.881764] pqi_irq_handler+0x1fc/0x13f0 [smartpqi] [ 9371.914115] __handle_irq_event_percpu+0x81/0x550 [ 9371.924289] handle_irq_event_percpu+0x70/0x100 [ 9371.945559] handle_irq_event+0x5a/0x8b [ 9371.950121] handle_edge_irq+0x10c/0x370 [ 9371.950121] handle_edge_irq+0x10c/0x370 [ 9371.959858] asm_call_on_stack+0x12/0x20 asm_call_on_stack at arch/x86/entry/entry_64.S:710 [ 9371.964899] </IRQ> [ 9371.967716] common_interrupt+0x185/0x2a0 [ 9371.972455] asm_common_interrupt+0x1e/0x40 [ 9371.977368] RIP: 0010:__asan_load4+0x8/0xa0 [ 9371.982281] Code: 00 e8 5c f4 ff ff 5d c3 40 38 f0 0f 9e c0 84 c0 75 e5 5d c3 48 c1 e8 03 80 3c 10 00 75 ed 5d c3 66 90 55 48 89 e5 48 8b 4d 08 <48> 83 ff fb 77 6c eb 3a 0f 1f 00 48 b8 00 00 00 00 00 00 00 ff 48 [ 9372.002246] RSP: 0018:ffffc9000b12f0e8 EFLAGS: 00000202 [ 9372.008207] RAX: 0000000000000000 RBX: ffffc9000b12f150 RCX: ffffffff9ed32487 [ 9372.016489] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffffffff9ff92a94 [ 9372.024370] RBP: ffffc9000b12f0e8 R08: fffffbfff3ff1d4d R09: fffffbfff3ff1d4d [ 9372.032252] R10: ffffffff9ff8ea67 R11: fffffbfff3ff1d4c R12: 0000000000000000 [ 9372.040490] R13: ffffc9000b12f358 R14: 000000000000000c R15: 0000000000000005 [ 9372.053899] debug_lockdep_rcu_enabled+0x27/0x60 [ 9372.059248] rcu_read_lock_held_common+0x12/0x60 [ 9372.064989] rcu_read_lock_sched_held+0x60/0xe0 [ 9372.080338] shrink_active_list+0xbfd/0xc30 [ 9372.120481] shrink_lruvec+0xbf1/0x11b0 [ 9372.150410] shrink_node+0x344/0xd10 [ 9372.154719] do_try_to_free_pages+0x263/0xa00 [ 9372.169860] try_to_free_pages+0x239/0x570 [ 9372.179356] __alloc_pages_slowpath.constprop.59+0x5dd/0x1880 [ 9372.215762] __alloc_pages_nodemask+0x562/0x670 [ 9372.232686] alloc_pages_current+0x9c/0x110 [ 9372.237599] alloc_slab_page+0x355/0x530 [ 9372.242755] allocate_slab+0x485/0x5a0 [ 9372.247232] new_slab+0x46/0x70 [ 9372.251095] ___slab_alloc+0x35f/0x810 [ 9372.274485] __slab_alloc+0x43/0x70 [ 9372.287650] kmem_cache_alloc+0x257/0x3d0 [ 9372.298182] prepare_creds+0x26/0x130 [ 9372.302571] do_faccessat+0x255/0x3e0 [ 9372.321591] __x64_sys_access+0x38/0x40 [ 9372.326154] do_syscall_64+0x64/0x340 [ 9372.330542] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 9372.336329] RIP: 0033:0x7fc983a21bfb [ 9372.341050] Code: Bad RIP value. [ 9372.345000] RSP: 002b:00007fffd543cde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000015 [ 9372.353319] RAX: ffffffffffffffda RBX: 00007fffd543fa40 RCX: 00007fc983a21bfb [ 9372.361199] RDX: 00007fc983cf1c00 RSI: 0000000000000000 RDI: 00005573ed458090 [ 9372.369512] RBP: 00007fffd543cf30 R08: 00005573ed44ab99 R09: 0000000000000007 [ 9372.377393] R10: 0000000000000041 R11: 0000000000000246 R12: 0000000000000000 [ 9372.385275] R13: 0000000000000000 R14: 00007fffd543cea0 R15: 00005573eecd2990 [ 9372.393608] irq event stamp: 27274496 [ 9372.397999] hardirqs last enabled at (27274495): [<ffffffff9e635b8e>] free_unref_page_list+0x2ee/0x400 [ 9372.408152] hardirqs last disabled at (27274496): [<ffffffff9ed2c22b>] idtentry_enter_cond_rcu+0x1b/0x50 [ 9372.418695] softirqs last enabled at (27272816): [<ffffffff9f000478>] __do_softirq+0x478/0x784 [ 9372.428154] softirqs last disabled at (27272807): [<ffffffff9e2d0b41>] irq_exit_rcu+0xd1/0xe0 [ 9372.437435] ---[ end trace d2ebac1fad6e452e ]---
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Qian,
can you please ensure that people who got cc'ed because the problem affects their subsystem are included on your replies even if you are replying to a different subthread?
I explicitely did:
at the very beginning of my reply:
https://lore.kernel.org/r/[email protected]
to make you aware of that.
Yes, email sucks, but it sucks even more when people are careless.
Quoted 4 lines · Qian Cai
On Fri, Jun 05, 2020 at 07:36:22PM +0200, Peter Zijlstra wrote:
Even after I trimmed the .config [1] to the barely minimal, a subset of LTP test still unable to finish on those AMD servers with page_owner=on.
What a surprise...
Quoted 12 lines
[1]:
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.configIt looks like because this new IRQ entry introduced by this patch,
</IRQ>
asm_call_on_stackatarch/x86/entry/entry_64.S:710handle_edge_irqatkernel/irq/chip.c:832which will running out of the stack depot limit due to nested loops below.
Quoted 8 lines
which has this loop,
do { ... handle_irq_event(desc); ... } while ((desc->istate & IRQS_PENDING) && !irqd_irq_disabled(&desc->irq_data));
This loop has absolutely nothing to do with stack entry usage.
foo()
do {
bar(); } while (condition);
}
If you take a stack trace inside bar() it will be the same stack trace for every single loop iteration. And that stack trace will not be any different from:
foo()
{
bar():
}
assumed that the call chain leading to foo() is the same in both cases.
And you can add even more loops in subsequent call chains within bar(). They do not matter at all.
Quoted 7 lines
Here has a nested loop,
for_each_action_of_desc(desc, action) { ... res = action->handler(irq, action->dev_id); ... }
And this one is completely irrelevant because the interrupt which we are looking at is a PCI interrupt which CANNOT be shared. IOW, the number of loop iterations and the number of handlers invoked is exactly ONE.
I seriously have no idea what you are trying to demonstrate by finding loops in a SINGLE callchain.
Quoted 10 lines
Here has one more nested loop,
while (table->orig_nents) { ... free_fn(sgl, alloc_size); ... }
free_fn()will callkmem_cache_free(). Since we have page_owner=on, it will callsave_stack()to save the each free stack trace.
That stack trace for each invocation of free_fn() in this loop is exactly the same stack trace. The same stack trace is not eating up any memory because the hash matches, i.e. the stack trace in the depot is already known.
Here is the simplified difference between the old code and the new code:
Old New
handle_edge_irq handle_edge_irq
do_IRQ asm_call_on_stack
common_interrupt common_interrupt
asm_common_interrupt
IOW, for every _UNIQUE_ interrupt related call chain, there is exactly ONE stack entry more than before.
For a loop which generates the exact same stack trace for every iteration this extra entry is not a problem.
But what matters is that interrupts can hit any random code path. So the amount of possible non-unique call chains is pretty much unlimited. And with a high number of non-unique call chains the extra entry starts to matter.
It's trival math, isn't it?
TS = Total Size of depot as = average size of all stored unique stack traces ms = maximum number of unqiue stack traces which fit in TS
ms_old = TS / as
Lets further assume that the vast majority of stack traces are taken from interrupt context. That means with the new code this results in:
ms_new = TS / (as + 1)
==> ms_new = ms_old * as / (as + 1)
Depending on the value of 'as' the +1 can shave off a significant percentage of capacity. IOW, the capacity is simply too small now for the test scenario you are running. Truly a surprising outcome, right?
To get facts instead of useless loop theories, can you please apply the patch below, enable DEBUGFS and provide the output of
/sys/kernel/debug/stackdepot/info
for a kernel before that change and after? Please read out that file at periodically roughly the same amounts of time after starting your test scenario.
Note, that I doubled the size of the stack depot so that we get real numbers and not the cutoff by the size limit. IOW, the warning should not trigger anymore. If it triggers nevertheless then the numbers will still tell us an interesting story.
Thanks,
tglx
--- lib/stackdepot.c | 43 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-)
lib/stackdepot.c +42 -1
@@ -31,6 +31,7 @@#include <linux/stackdepot.h> #include <linux/string.h> #include <linux/types.h> +#include <linux/debugfs.h> #define DEPOT_STACK_BITS (sizeof(depot_stack_handle_t) * 8)
@@ -42,7 +43,7 @@ STACK_ALLOC_ALIGN)#define STACK_ALLOC_INDEX_BITS (DEPOT_STACK_BITS - \ STACK_ALLOC_NULL_PROTECTION_BITS - STACK_ALLOC_OFFSET_BITS) -#define STACK_ALLOC_SLABS_CAP 8192 +#define STACK_ALLOC_SLABS_CAP 16384#define STACK_ALLOC_MAX_SLABS \ (((1LL << (STACK_ALLOC_INDEX_BITS)) < STACK_ALLOC_SLABS_CAP) ? \ (1LL << (STACK_ALLOC_INDEX_BITS)) : STACK_ALLOC_SLABS_CAP)
@@ -70,6 +71,7 @@ static void *stack_slabs[STACK_ALLOC_MAXstatic int depot_index;static int next_slab_inited;static size_t depot_offset; +static unsigned long unique_stacks;static DEFINE_SPINLOCK (depot_lock);static bool init_stack_slab (void **prealloc)
@@ -138,6 +140,7 @@ static struct stack_record *depot_alloc_ stack->handle.valid =1 ;memcpy (stack->entries, entries, size *sizeof (unsigned long )); depot_offset += required_size; + unique_stacks++;return stack; }
@@ -340,3 +343,41 @@ unsigned int filter_irq_stacks(unsignedreturn nr_entries; }EXPORT_SYMBOL_GPL (filter_irq_stacks); + +static int debug_show (struct seq_file *m,void *p) +{ +unsigned long unst; +int didx, doff; + +spin_lock_irq (&depot_lock); + unst = unique_stacks; + didx = depot_index; + doff = depot_offset; +spin_unlock_irq (&depot_lock); + +seq_printf (m,"Unique stacks: %lu\n" , unst); +seq_printf (m,"Depot index: %d\n" , didx); +seq_printf (m,"Depot offset: %d\n" , doff); +return 0 ; +} + +static int debug_open (struct inode *inode,struct file *file) +{ +return single_open (file, debug_show, inode->i_private); +} + +static const struct file_operations dfs_ops = { + .open = debug_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +static int __init debugfs_init (void ) +{ +struct dentry *root_dir =debugfs_create_dir ("stackdepot" ,NULL ); + +debugfs_create_file ("info" ,0444 , root_dir,NULL , &dfs_ops); +return 0 ; +} +__initcall (debugfs_init);
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 10 lines · Thomas Gleixner
Qian,
can you please ensure that people who got cc'ed because the problem affects their subsystem are included on your replies even if you are replying to a different subthread?
I explicitely did:
Cc + Alexander
Quoted 7 lines
at the very beginning of my reply:
https://lore.kernel.org/r/[email protected]to make you aware of that.
Yes, email sucks, but it sucks even more when people are careless.
Sorry, I will remeber that next time.
[]
Quoted 5 lines
To get facts instead of useless loop theories, can you please apply the patch below, enable DEBUGFS and provide the output of
/sys/kernel/debug/stackdepot/info
Quoted 8 lines
for a kernel before that change and after? Please read out that file at periodically roughly the same amounts of time after starting your test scenario.
Note, that I doubled the size of the stack depot so that we get real numbers and not the cutoff by the size limit. IOW, the warning should not trigger anymore. If it triggers nevertheless then the numbers will still tell us an interesting story.
Instead of running the whole testsuite, I just picked this single LTP oom02 test which seems usually trigger it within the testsuite. Let me know if this is insufficient (which indeed tell the big difference in "Unique stacks"), and I am happy to run the whole things.
BAD: next-20200608
GOOD: next-20200528 (which does not include this series)
BAD (after boot)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 33547 Depot index: 359 Depot offset: 6752
BAD (after oom02)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 140476 Depot index: 2555 Depot offset: 9168
GOOD (after boot)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 31112 Depot index: 317 Depot offset: 14384
GOOD (after oom02)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 34176 Depot index: 354 Depot offset: 4032
BTW, I am happy to run another one using next-20200608 with just this series reverted if you suspect there is something else going on between those two trees.
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Qian,
Quoted 10 lines · Qian Cai
Quoted 4 lines · Thomas Gleixner
Note, that I doubled the size of the stack depot so that we get real numbers and not the cutoff by the size limit. IOW, the warning should not trigger anymore. If it triggers nevertheless then the numbers will still tell us an interesting story.
Instead of running the whole testsuite, I just picked this single LTP oom02 test which seems usually trigger it within the testsuite. Let me know if this is insufficient (which indeed tell the big difference in "Unique stacks"), and I am happy to run the whole things.
thanks for providing the data.
Quoted 4 lines
BAD: next-20200608
GOOD: next-20200528 (which does not include this series)BAD (after boot)
Quoted 8 lines
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 33547 Depot index: 359 Depot offset: 6752BAD (after oom02)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 140476
That's indeed odd. I try to reproduce and figure out what really breaks here.
Thanks,
tglx
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Qian,
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines
BAD (after oom02)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 140476That's indeed odd. I try to reproduce and figure out what really breaks here.
I checked your config file and I think I know where this comes from. Can you plase disable KASAN just for testing purposes and compare before after again?
Thanks,
tglx
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 7 lines · Thomas Gleixner
Qian,
Quoted 4 lines · Thomas Gleixner
Quoted 3 lines
BAD (after oom02)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 140476
Quoted 6 lines
Quoted 2 lines
That's indeed odd. I try to reproduce and figure out what really breaks here.
I checked your config file and I think I know where this comes from. Can you plase disable KASAN just for testing purposes and compare before after again?
It turns out I'll need a few days to be able to get ahold of those affected systems again. I'll be reporting back as soon as possible.
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 9 lines · Qian Cai
Quoted 8 lines · Thomas Gleixner
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines
BAD (after oom02)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 140476That's indeed odd. I try to reproduce and figure out what really breaks here.
Quoted 6 lines
Quoted 3 lines
I checked your config file and I think I know where this comes from. Can you plase disable KASAN just for testing purposes and compare before after again?
It turns out I'll need a few days to be able to get ahold of those affected systems again. I'll be reporting back as soon as possible.
I figured it out. It has nothing to do with asm_call_on...(). It's also unrelated to page_owner. It's purely a KASAN issue.
KASAN trims stack traces with the help of filter_irq_stacks() which relies on __irqentry_text_start/end
The entry rework moved the interrupt entry points out of the irqentry section, which breaks that filter function.
I made it work again. Here are the numbers which make that obvious:
Mainline:
Unique stacks: 23403 Depot index: 244 Depot offset: 4208
x86/entry:
Unique stacks: 38835 Depot index: 464 Depot offset: 7616
x86/entry + fix:
Unique stacks: 23607 Depot index: 247 Depot offset: 14224
So with the non-working trimming this generates more unique stacks and because they are not trimmed they become larger and eat more storage space. The resulting average per stack shows that:
Mainline: 171 bytes per stack x86/entry: 195 bytes per stack
x86/entry + fix: 172 bytes per stack
I'll point you to a test branch shortly.
Thanks,
tglx
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 47 lines · Thomas Gleixner
Quoted 9 lines · Qian Cai
Quoted 8 lines · Thomas Gleixner
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines
BAD (after oom02)
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 140476That's indeed odd. I try to reproduce and figure out what really breaks here.
Quoted 6 lines
Quoted 3 lines
I checked your config file and I think I know where this comes from. Can you plase disable KASAN just for testing purposes and compare before after again?
It turns out I'll need a few days to be able to get ahold of those affected systems again. I'll be reporting back as soon as possible.
I figured it out. It has nothing to do with asm_call_on...(). It's also unrelated to
page_owner. It's purely a KASAN issue.KASAN trims stack traces with the help of
filter_irq_stacks()which relies on __irqentry_text_start/endThe entry rework moved the interrupt entry points out of the irqentry section, which breaks that filter function.
I made it work again. Here are the numbers which make that obvious:
Mainline:
Unique stacks: 23403 Depot index: 244 Depot offset: 4208x86/entry:
Unique stacks: 38835 Depot index: 464 Depot offset: 7616x86/entry + fix:
Unique stacks: 23607 Depot index: 247 Depot offset: 14224So with the non-working trimming this generates more unique stacks and because they are not trimmed they become larger and eat more storage space. The resulting average per stack shows that:
Quoted 5 lines
Mainline: 171 bytes per stack x86/entry: 195 bytes per stackx86/entry + fix: 172 bytes per stack
I'll point you to a test branch shortly.
Thomas, I get ahold of one of the affected systems now if you want some testing there.
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Qian,
Quoted 4 lines · Qian Cai
On Wed, Jun 10, 2020 at 09:38:56PM +0200, Thomas Gleixner wrote:
Thomas, I get ahold of one of the affected systems now if you want some testing there.
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry
Thanks,
tglx
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 9 lines · Thomas Gleixner
Qian,
Quoted 4 lines · Qian Cai
On Wed, Jun 10, 2020 at 09:38:56PM +0200, Thomas Gleixner wrote:
Thomas, I get ahold of one of the affected systems now if you want some testing there.
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitx86/entry
Looks good.
<start>
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 27481 Depot index: 285 Depot offset: 3024
<after oom02>
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 29242 Depot index: 308 Depot offset: 6608
<after the full testsuite>
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 49221 Depot index: 530 Depot offset: 5888
Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack
Quoted 25 lines · Qian Cai
Quoted 9 lines · Thomas Gleixner
Qian,
Quoted 4 lines · Qian Cai
On Wed, Jun 10, 2020 at 09:38:56PM +0200, Thomas Gleixner wrote:
Thomas, I get ahold of one of the affected systems now if you want some testing there.
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.gitx86/entryLooks good.
<start>
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 27481 Depot index: 285 Depot offset: 3024<after oom02>
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 29242 Depot index: 308 Depot offset: 6608
Quoted 5 lines
<after the full testsuite>
# cat /sys/kernel/debug/stackdepot/info Unique stacks: 49221 Depot index: 530 Depot offset: 5888
Thanks for confirming!
PATCH v9 11/39 x86/entry/64: Move do_softirq_own_stack() to C
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
The first step to get rid of the ENTER/LEAVE_IRQ_STACK ASM macro maze. Use the new C code helpers to move do_softirq_own_stack() out of ASM code.
---
V9: Simplified version
--- arch/x86/entry/entry_64.S | 13 ------------- arch/x86/kernel/irq_64.c | 6 ++++++ 2 files changed, 6 insertions(+), 13 deletions(-)
arch/x86/entry/entry_64.S +0 -13
@@ -1145,19 +1145,6 @@ SYM_FUNC_START(asm_call_on_stack) ret SYM_FUNC_END(asm_call_on_stack) -/* Call softirq on interrupt stack. Interrupts are off. */ -.pushsection .text, "ax" -SYM_FUNC_START(do_softirq_own_stack) - pushq %rbp - mov %rsp, %rbp - ENTER_IRQ_STACK regs=0 old_rsp=%r11 - call __do_softirq - LEAVE_IRQ_STACK regs=0 - leaveq - ret -SYM_FUNC_END(do_softirq_own_stack) -.popsection - #ifdef CONFIG_XEN_PV /* * A note on the "critical region" in our callback handler.
arch/x86/kernel/irq_64.c +6 -0
@@ -20,6 +20,7 @@#include <linux/sched/task_stack.h> #include <asm/cpu_entry_area.h> +#include <asm/irq_stack.h> #include <asm/io_apic.h> #include <asm/apic.h>
@@ -70,3 +71,8 @@ int irq_init_percpu_irqstack(unsigned inreturn 0 ;return map_irq_stack (cpu); } + +void do_softirq_own_stack (void ) +{ +run_on_irqstack_cond (__do_softirq,NULL ,NULL ); +}
[tip: x86/entry] x86/entry/64: Move do_softirq_own_stack() to C
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 4f41db23ef2593aec26e23e7f4bacc620e82f545 Gitweb: https://git.kernel.org/tip/4f41db23ef2593aec26e23e7f4bacc620e82f545 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:24 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
x86/entry/64: Move do_softirq_own_stack() to C
The first step to get rid of the ENTER/LEAVE_IRQ_STACK ASM macro maze. Use the new C code helpers to move do_softirq_own_stack() out of ASM code.
--- arch/x86/entry/entry_64.S | 13 ------------- arch/x86/kernel/irq_64.c | 6 ++++++ 2 files changed, 6 insertions(+), 13 deletions(-)
arch/x86/entry/entry_64.S +0 -13
@@ -1145,19 +1145,6 @@ SYM_FUNC_START(asm_call_on_stack) ret SYM_FUNC_END(asm_call_on_stack) -/* Call softirq on interrupt stack. Interrupts are off. */ -.pushsection .text, "ax" -SYM_FUNC_START(do_softirq_own_stack) - pushq %rbp - mov %rsp, %rbp - ENTER_IRQ_STACK regs=0 old_rsp=%r11 - call __do_softirq - LEAVE_IRQ_STACK regs=0 - leaveq - ret -SYM_FUNC_END(do_softirq_own_stack) -.popsection - #ifdef CONFIG_XEN_PV /* * A note on the "critical region" in our callback handler.
arch/x86/kernel/irq_64.c +6 -0
@@ -20,6 +20,7 @@#include <linux/sched/task_stack.h> #include <asm/cpu_entry_area.h> +#include <asm/irq_stack.h> #include <asm/io_apic.h> #include <asm/apic.h>
@@ -70,3 +71,8 @@ int irq_init_percpu_irqstack(unsigned int cpu)return 0 ;return map_irq_stack (cpu); } + +void do_softirq_own_stack (void ) +{ +run_on_irqstack_cond (__do_softirq,NULL ,NULL ); +}
PATCH v9 12/39 x86/entry: Split out idtentry_exit_cond_resched()
In reply to Thomas GleixnerThe XEN PV hypercall requires the ability of conditional rescheduling when preemption is disabled because some hypercalls take ages.
Split out the rescheduling code from idtentry_exit_cond_rcu() so it can be reused for that.
---
V9: New patch
--- arch/x86/entry/common.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
arch/x86/entry/common.c +15 -15
@@ -580,6 +580,20 @@ bool noinstr idtentry_enter_cond_rcu(strreturn false; } +static void idtentry_exit_cond_resched (struct pt_regs *regs,bool may_sched) +{ +if (may_sched && !preempt_count ()) { +/* Sanity check RCU and thread stack */ +rcu_irq_exit_check_preempt (); +if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) +WARN_ON_ONCE (!on_thread_stack ()); +if (need_resched ()) +preempt_schedule_irq (); + } +/* Covers both tracing and lockdep */ +trace_hardirqs_on (); +} +/** * idtentry_exit_cond_rcu - Handle return from exception with conditional RCU * handling
@@ -621,21 +635,7 @@ void noinstr idtentry_exit_cond_rcu(stru }instrumentation_begin (); - -/* Check kernel preemption, if enabled */ -if (IS_ENABLED (CONFIG_PREEMPTION )) { -if (!preempt_count ()) { -/* Sanity check RCU and thread stack */ -rcu_irq_exit_check_preempt (); -if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) -WARN_ON_ONCE (!on_thread_stack ()); -if (need_resched ()) -preempt_schedule_irq (); - } - } -/* Covers both tracing and lockdep */ -trace_hardirqs_on (); - +idtentry_exit_cond_resched (regs,IS_ENABLED (CONFIG_PREEMPTION ));instrumentation_end (); }else {/*
[tip: x86/entry] x86/entry: Split out idtentry_exit_cond_resched()
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 08f086303b4eddc61da264c2e9a8a5ee7f260604 Gitweb: https://git.kernel.org/tip/08f086303b4eddc61da264c2e9a8a5ee7f260604 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:25 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
x86/entry: Split out idtentry_exit_cond_resched()
The XEN PV hypercall requires the ability of conditional rescheduling when preemption is disabled because some hypercalls take ages.
Split out the rescheduling code from idtentry_exit_cond_rcu() so it can be reused for that.
--- arch/x86/entry/common.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
arch/x86/entry/common.c +15 -15
@@ -583,6 +583,20 @@ bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs)return false; } +static void idtentry_exit_cond_resched (struct pt_regs *regs,bool may_sched) +{ +if (may_sched && !preempt_count ()) { +/* Sanity check RCU and thread stack */ +rcu_irq_exit_check_preempt (); +if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) +WARN_ON_ONCE (!on_thread_stack ()); +if (need_resched ()) +preempt_schedule_irq (); + } +/* Covers both tracing and lockdep */ +trace_hardirqs_on (); +} +/** * idtentry_exit_cond_rcu - Handle return from exception with conditional RCU * handling
@@ -624,21 +638,7 @@ void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit) }instrumentation_begin (); - -/* Check kernel preemption, if enabled */ -if (IS_ENABLED (CONFIG_PREEMPTION )) { -if (!preempt_count ()) { -/* Sanity check RCU and thread stack */ -rcu_irq_exit_check_preempt (); -if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) -WARN_ON_ONCE (!on_thread_stack ()); -if (need_resched ()) -preempt_schedule_irq (); - } - } -/* Covers both tracing and lockdep */ -trace_hardirqs_on (); - +idtentry_exit_cond_resched (regs,IS_ENABLED (CONFIG_PREEMPTION ));instrumentation_end (); }else {/*
PATCH v9 13/39 x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Convert the XEN/PV hypercall to IDTENTRY:
- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Remove the old prototypes
The handler stubs need to stay in ASM code as it needs corner case handling and adjustment of the stack pointer.
Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack if required.
The exit code is slightly different from the regular idtentry_exit() on non-preemptible kernels. If the hypercall is preemptible and need_resched() is set then XEN provides a preempt hypercall scheduling function.
Move this functionality into the entry code so it can use the existing idtentry functionality.
---
V9: Simplified the schedule magic
--- arch/x86/entry/common.c | 72 ++++++++++++++++++++++++++++++++++++++++ arch/x86/entry/entry_32.S | 17 +++++---- arch/x86/entry/entry_64.S | 22 +++--------- arch/x86/include/asm/idtentry.h | 13 +++++++ arch/x86/xen/setup.c | 4 +- arch/x86/xen/smp_pv.c | 3 + arch/x86/xen/xen-asm_32.S | 12 +++--- arch/x86/xen/xen-asm_64.S | 2 - arch/x86/xen/xen-ops.h | 1 drivers/xen/Makefile | 2 - drivers/xen/preempt.c | 42 ----------------------- 11 files changed, 115 insertions(+), 75 deletions(-)
arch/x86/entry/common.c +72 -0
@@ -27,6 +27,9 @@#include <linux/syscalls.h> #include <linux/uaccess.h> +#include <xen/xen-ops.h> +#include <xen/events.h> +#include <asm/desc.h> #include <asm/traps.h> #include <asm/vdso.h>
@@ -35,6 +38,7 @@#include <asm/nospec-branch.h> #include <asm/io_bitmap.h> #include <asm/syscall.h> +#include <asm/irq_stack.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
@@ -677,3 +681,71 @@ void noinstr idtentry_exit_user(struct pprepare_exit_to_usermode (regs); } + +#ifdef CONFIG_XEN_PV + +#ifndef CONFIG_PREEMPTION +/* +* Some hypercalls issued by the toolstack can take many 10s of +* seconds. Allow tasks running hypercalls via the privcmd driver to +* be voluntarily preempted even if full kernel preemption is +* disabled. +* +* Such preemptible hypercalls are bracketed by +* xen_preemptible_hcall_begin() and xen_preemptible_hcall_end() +* calls. +*/ +DEFINE_PER_CPU (bool, xen_in_preemptible_hcall); +EXPORT_SYMBOL_GPL (xen_in_preemptible_hcall); + +/* +* In case of scheduling the flag must be cleared and restored after +* returning from schedule as the task might move to a different CPU. +*/ +static __always_inline bool get_and_clear_inhcall(void) +{ +boot inhcall =__this_cpu_read (xen_in_preemptible_hcall); + +__this_cpu_write (xen_in_preemptible_hcall, false); +} + +static __always_inline void restore_inhcall(bool inhcall) +{ +__this_cpu_write (xen_in_preemptible_hcall, inhcall); +} +#else +static __always_inline bool get_and_clear_inhcall(void) { return false; } +static __always_inline void restore_inhcall(bool inhcall) { } +#endif + +static void __xen_pv_evtchn_do_upcall (void ) +{ +irq_enter_rcu (); +inc_irq_stat (irq_hv_callback_count); + +xen_hvm_evtchn_do_upcall (); + +irq_exit_rcu (); +} + +__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +bool inhcall, rcu_exit; + + rcu_exit =idtentry_enter_cond_rcu (regs); + old_regs =set_irq_regs (regs); + +run_on_irqstack_cond (__xen_pv_evtchn_do_upcall,NULL , regs); + +set_irq_regs (old_regs); + + inhcall =get_and_clear_inhcall (); +if (inhcall && !WARN_ON_ONCE (rcu_exit)) { +idtentry_exit_cond_resched (regs, true); +restore_inhcall (inhcall); + }else { +idtentry_exit_cond_rcu (regs, rcu_exit); + } +} +#endif /* CONFIG_XEN_PV */
arch/x86/entry/entry_32.S +9 -8
@@ -1298,7 +1298,10 @@ SYM_CODE_END(native_iret) #endif #ifdef CONFIG_XEN_PV -SYM_FUNC_START(xen_hypervisor_callback) +/* + * See comment in entry_64.S for further explanation + */ +SYM_FUNC_START(exc_xen_hypervisor_callback) /* * Check to see if we got the event in the critical * region in xen_iret_direct, after we've reenabled
@@ -1315,14 +1318,11 @@ SYM_FUNC_START(xen_hypervisor_callback) pushl $-1 /* orig_ax = -1 => not a system call */ SAVE_ALL ENCODE_FRAME_POINTER - TRACE_IRQS_OFF + mov %esp, %eax - call xen_evtchn_do_upcall -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp ret_from_intr -SYM_FUNC_END(xen_hypervisor_callback) + call xen_pv_evtchn_do_upcall + jmp handle_exception_return +SYM_FUNC_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
@@ -1464,6 +1464,7 @@ SYM_CODE_START_LOCAL_NOALIGN(handle_exce movl %esp, %eax # pt_regs pointer CALL_NOSPEC edi +handle_exception_return: #ifdef CONFIG_VM86 movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS movb PT_CS(%esp), %al
arch/x86/entry/entry_64.S +7 -15
@@ -1067,10 +1067,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1 -#ifdef CONFIG_XEN_PV -idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0 -#endif - /* * Reload gs selector with exception handling * edi: new selector
@@ -1158,9 +1154,10 @@ SYM_FUNC_END(asm_call_on_stack) * So, on entry to the handler we detect whether we interrupted an * existing activation in its critical region -- if so, we pop the current * activation and restart the handler using the previous one. + * + * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs) */ -/* do_hypervisor_callback(struct *pt_regs) */ -SYM_CODE_START_LOCAL(xen_do_hypervisor_callback) +SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) /* * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1170,15 +1167,10 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_c movq %rdi, %rsp /* we don't return, adjust the stack frame */ UNWIND_HINT_REGS - ENTER_IRQ_STACK old_rsp=%r10 - call xen_evtchn_do_upcall - LEAVE_IRQ_STACK - -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp error_exit -SYM_CODE_END(xen_do_hypervisor_callback) + call xen_pv_evtchn_do_upcall + + jmp error_return +SYM_CODE_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
arch/x86/include/asm/idtentry.h +13 -0
@@ -337,6 +337,13 @@ static __always_inline void __##func(str* This avoids duplicate defines and ensures that everything is consistent. */ +/* +* Dummy trap number so the low level ASM macro vector number checks do not +* match which results in emitting plain IDTENTRY stubs without bells and +* whistels. +*/ +#define X86_TRAP_OTHER 0xFFFF +/* Simple exception entry points. No hardware error code */ DECLARE_IDTENTRY (X86_TRAP_DE , exc_divide_error);DECLARE_IDTENTRY (X86_TRAP_OF , exc_overflow);
@@ -376,4 +383,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB, debug)/* #DF */ DECLARE_IDTENTRY_DF (X86_TRAP_DF , exc_double_fault); +#ifdef CONFIG_XEN_PV +DECLARE_IDTENTRY (X86_TRAP_OTHER , exc_xen_hypervisor_callback); +#endif + +#undef X86_TRAP_OTHER +#endif
arch/x86/xen/setup.c +3 -1
@@ -20,6 +20,7 @@#include <asm/setup.h> #include <asm/acpi.h> #include <asm/numa.h> +#include <asm/idtentry.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h>
@@ -993,7 +994,8 @@ static void __init xen_pvmmu_arch_setup(HYPERVISOR_vm_assist (VMASST_CMD_enable, VMASST_TYPE_pae_extended_cr3); -if (register_callback (CALLBACKTYPE_event, xen_hypervisor_callback) || +if (register_callback (CALLBACKTYPE_event, + xen_asm_exc_xen_hypervisor_callback) ||register_callback (CALLBACKTYPE_failsafe, xen_failsafe_callback))BUG ();
arch/x86/xen/smp_pv.c +2 -1
@@ -27,6 +27,7 @@#include <asm/paravirt.h> #include <asm/desc.h> #include <asm/pgtable.h> +#include <asm/idtentry.h> #include <asm/cpu.h> #include <xen/interface/xen.h>
@@ -347,7 +348,7 @@ cpu_initialize_context(unsigned int cpu, ctxt->gs_base_kernel =per_cpu_offset (cpu);#endif ctxt->event_callback_eip = - (unsigned long )xen_hypervisor_callback; + (unsigned long )xen_asm_exc_xen_hypervisor_callback; ctxt->failsafe_callback_eip = (unsigned long )xen_failsafe_callback;per_cpu (xen_cr3, cpu) =__pa (swapper_pg_dir);
arch/x86/xen/xen-asm_32.S +7 -5
@@ -93,7 +93,7 @@ SYM_CODE_START(xen_iret) /* * If there's something pending, mask events again so we can - * jump back into xen_hypervisor_callback. Otherwise do not + * jump back into exc_xen_hypervisor_callback. Otherwise do not * touch XEN_vcpu_info_mask. */ jne 1f
@@ -113,7 +113,7 @@ SYM_CODE_START(xen_iret) * Events are masked, so jumping out of the critical region is * OK. */ - je xen_hypervisor_callback + je asm_exc_xen_hypervisor_callback 1: iret xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret) .globl xen_iret_start_crit, xen_iret_end_crit /* - * This is called by xen_hypervisor_callback in entry_32.S when it sees + * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees * that the EIP at the time of interrupt was between * xen_iret_start_crit and xen_iret_end_crit. *
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret) * eflags } * cs } nested exception info * eip } - * return address : (into xen_hypervisor_callback) + * return address : (into asm_exc_xen_hypervisor_callback) * * In order to deliver the nested exception properly, we need to discard the * nested exception frame such that when we handle the exception, we do it
@@ -152,7 +152,8 @@ SYM_CODE_END(xen_iret) * * The only caveat is that if the outer eax hasn't been restored yet (i.e. * it's still on stack), we need to restore its value here. - */ +*/ +.pushsection .noinstr.text, "ax" SYM_CODE_START(xen_iret_crit_fixup) /* * Paranoia: Make sure we're really coming from kernel space.
@@ -181,3 +182,4 @@ SYM_CODE_START(xen_iret_crit_fixup) 2: ret SYM_CODE_END(xen_iret_crit_fixup) +.popsection
arch/x86/xen/xen-asm_64.S +1 -1
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_err #ifdef CONFIG_IA32_EMULATION xen_pv_trap entry_INT80_compat #endif -xen_pv_trap hypervisor_callback +xen_pv_trap asm_exc_xen_hypervisor_callback __INIT SYM_CODE_START(xen_early_idt_handler_array)
arch/x86/xen/xen-ops.h +0 -1
@@ -8,7 +8,6 @@#include <xen/xen-ops.h> /* These are code, but not functions. Defined in entry.S */ -extern const char xen_hypervisor_callback[];extern const char xen_failsafe_callback[];void xen_sysenter_target (void );
drivers/xen/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_HOTPLUG_CPU) += cpu_hotplug.o -obj-y += grant-table.o features.o balloon.o manage.o preempt.o time.o +obj-y += grant-table.o features.o balloon.o manage.o time.o obj-y += mem-reservation.o obj-y += events/ obj-y += xenbus/
drivers/xen/preempt.c deleted +0 -42
42 lines not on this page · raw message
PATCH v9 13/39 x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Convert the XEN/PV hypercall to IDTENTRY:
- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Remove the old prototypes
The handler stubs need to stay in ASM code as it needs corner case handling and adjustment of the stack pointer.
Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack if required.
The exit code is slightly different from the regular idtentry_exit() on non-preemptible kernels. If the hypercall is preemptible and need_resched() is set then XEN provides a preempt hypercall scheduling function.
Move this functionality into the entry code so it can use the existing idtentry functionality.
---
V9-1: Fixes the fatfingered inline V9: Simplified the schedule magic --- arch/x86/entry/common.c | 73 ++++++++++++++++++++++++++++++++++++++++ arch/x86/entry/entry_32.S | 17 ++++----- arch/x86/entry/entry_64.S | 22 +++--------- arch/x86/include/asm/idtentry.h | 13 +++++++ arch/x86/xen/setup.c | 4 +- arch/x86/xen/smp_pv.c | 3 + arch/x86/xen/xen-asm_32.S | 12 +++--- arch/x86/xen/xen-asm_64.S | 2 - arch/x86/xen/xen-ops.h | 1 drivers/xen/Makefile | 2 - drivers/xen/preempt.c | 42 ----------------------- 11 files changed, 116 insertions(+), 75 deletions(-)
arch/x86/entry/common.c +73 -0
@@ -27,6 +27,9 @@#include <linux/syscalls.h> #include <linux/uaccess.h> +#include <xen/xen-ops.h> +#include <xen/events.h> +#include <asm/desc.h> #include <asm/traps.h> #include <asm/vdso.h>
@@ -35,6 +38,7 @@#include <asm/nospec-branch.h> #include <asm/io_bitmap.h> #include <asm/syscall.h> +#include <asm/irq_stack.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
@@ -677,3 +681,72 @@ void noinstr idtentry_exit_user(struct pprepare_exit_to_usermode (regs); } + +#ifdef CONFIG_XEN_PV + +#ifndef CONFIG_PREEMPTION +/* +* Some hypercalls issued by the toolstack can take many 10s of +* seconds. Allow tasks running hypercalls via the privcmd driver to +* be voluntarily preempted even if full kernel preemption is +* disabled. +* +* Such preemptible hypercalls are bracketed by +* xen_preemptible_hcall_begin() and xen_preemptible_hcall_end() +* calls. +*/ +DEFINE_PER_CPU (bool, xen_in_preemptible_hcall); +EXPORT_SYMBOL_GPL (xen_in_preemptible_hcall); + +/* +* In case of scheduling the flag must be cleared and restored after +* returning from schedule as the task might move to a different CPU. +*/ +static __always_inline bool get_and_clear_inhcall(void) +{ +bool inhcall =__this_cpu_read (xen_in_preemptible_hcall); + +__this_cpu_write (xen_in_preemptible_hcall, false); +return inhcall; +} + +static __always_inline void restore_inhcall(bool inhcall) +{ +__this_cpu_write (xen_in_preemptible_hcall, inhcall); +} +#else +static __always_inline bool get_and_clear_inhcall(void) { return false; } +static __always_inline void restore_inhcall(bool inhcall) { } +#endif + +static void __xen_pv_evtchn_do_upcall (void ) +{ +irq_enter_rcu (); +inc_irq_stat (irq_hv_callback_count); + +xen_hvm_evtchn_do_upcall (); + +irq_exit_rcu (); +} + +__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +bool inhcall, rcu_exit; + + rcu_exit =idtentry_enter_cond_rcu (regs); + old_regs =set_irq_regs (regs); + +run_on_irqstack_cond (__xen_pv_evtchn_do_upcall,NULL , regs); + +set_irq_regs (old_regs); + + inhcall =get_and_clear_inhcall (); +if (inhcall && !WARN_ON_ONCE (rcu_exit)) { +idtentry_exit_cond_resched (regs, true); +restore_inhcall (inhcall); + }else { +idtentry_exit_cond_rcu (regs, rcu_exit); + } +} +#endif /* CONFIG_XEN_PV */
arch/x86/entry/entry_32.S +9 -8
@@ -1298,7 +1298,10 @@ SYM_CODE_END(native_iret) #endif #ifdef CONFIG_XEN_PV -SYM_FUNC_START(xen_hypervisor_callback) +/* + * See comment in entry_64.S for further explanation + */ +SYM_FUNC_START(exc_xen_hypervisor_callback) /* * Check to see if we got the event in the critical * region in xen_iret_direct, after we've reenabled
@@ -1315,14 +1318,11 @@ SYM_FUNC_START(xen_hypervisor_callback) pushl $-1 /* orig_ax = -1 => not a system call */ SAVE_ALL ENCODE_FRAME_POINTER - TRACE_IRQS_OFF + mov %esp, %eax - call xen_evtchn_do_upcall -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp ret_from_intr -SYM_FUNC_END(xen_hypervisor_callback) + call xen_pv_evtchn_do_upcall + jmp handle_exception_return +SYM_FUNC_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
@@ -1464,6 +1464,7 @@ SYM_CODE_START_LOCAL_NOALIGN(handle_exce movl %esp, %eax # pt_regs pointer CALL_NOSPEC edi +handle_exception_return: #ifdef CONFIG_VM86 movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS movb PT_CS(%esp), %al
arch/x86/entry/entry_64.S +7 -15
@@ -1067,10 +1067,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1 -#ifdef CONFIG_XEN_PV -idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0 -#endif - /* * Reload gs selector with exception handling * edi: new selector
@@ -1158,9 +1154,10 @@ SYM_FUNC_END(asm_call_on_stack) * So, on entry to the handler we detect whether we interrupted an * existing activation in its critical region -- if so, we pop the current * activation and restart the handler using the previous one. + * + * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs) */ -/* do_hypervisor_callback(struct *pt_regs) */ -SYM_CODE_START_LOCAL(xen_do_hypervisor_callback) +SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) /* * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1170,15 +1167,10 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_c movq %rdi, %rsp /* we don't return, adjust the stack frame */ UNWIND_HINT_REGS - ENTER_IRQ_STACK old_rsp=%r10 - call xen_evtchn_do_upcall - LEAVE_IRQ_STACK - -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp error_exit -SYM_CODE_END(xen_do_hypervisor_callback) + call xen_pv_evtchn_do_upcall + + jmp error_return +SYM_CODE_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
arch/x86/include/asm/idtentry.h +13 -0
@@ -337,6 +337,13 @@ static __always_inline void __##func(str* This avoids duplicate defines and ensures that everything is consistent. */ +/* +* Dummy trap number so the low level ASM macro vector number checks do not +* match which results in emitting plain IDTENTRY stubs without bells and +* whistels. +*/ +#define X86_TRAP_OTHER 0xFFFF +/* Simple exception entry points. No hardware error code */ DECLARE_IDTENTRY (X86_TRAP_DE , exc_divide_error);DECLARE_IDTENTRY (X86_TRAP_OF , exc_overflow);
@@ -376,4 +383,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB, debug)/* #DF */ DECLARE_IDTENTRY_DF (X86_TRAP_DF , exc_double_fault); +#ifdef CONFIG_XEN_PV +DECLARE_IDTENTRY (X86_TRAP_OTHER , exc_xen_hypervisor_callback); +#endif + +#undef X86_TRAP_OTHER +#endif
arch/x86/xen/setup.c +3 -1
@@ -20,6 +20,7 @@#include <asm/setup.h> #include <asm/acpi.h> #include <asm/numa.h> +#include <asm/idtentry.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h>
@@ -993,7 +994,8 @@ static void __init xen_pvmmu_arch_setup(HYPERVISOR_vm_assist (VMASST_CMD_enable, VMASST_TYPE_pae_extended_cr3); -if (register_callback (CALLBACKTYPE_event, xen_hypervisor_callback) || +if (register_callback (CALLBACKTYPE_event, + xen_asm_exc_xen_hypervisor_callback) ||register_callback (CALLBACKTYPE_failsafe, xen_failsafe_callback))BUG ();
arch/x86/xen/smp_pv.c +2 -1
@@ -27,6 +27,7 @@#include <asm/paravirt.h> #include <asm/desc.h> #include <asm/pgtable.h> +#include <asm/idtentry.h> #include <asm/cpu.h> #include <xen/interface/xen.h>
@@ -347,7 +348,7 @@ cpu_initialize_context(unsigned int cpu, ctxt->gs_base_kernel =per_cpu_offset (cpu);#endif ctxt->event_callback_eip = - (unsigned long )xen_hypervisor_callback; + (unsigned long )xen_asm_exc_xen_hypervisor_callback; ctxt->failsafe_callback_eip = (unsigned long )xen_failsafe_callback;per_cpu (xen_cr3, cpu) =__pa (swapper_pg_dir);
arch/x86/xen/xen-asm_32.S +7 -5
@@ -93,7 +93,7 @@ SYM_CODE_START(xen_iret) /* * If there's something pending, mask events again so we can - * jump back into xen_hypervisor_callback. Otherwise do not + * jump back into exc_xen_hypervisor_callback. Otherwise do not * touch XEN_vcpu_info_mask. */ jne 1f
@@ -113,7 +113,7 @@ SYM_CODE_START(xen_iret) * Events are masked, so jumping out of the critical region is * OK. */ - je xen_hypervisor_callback + je asm_exc_xen_hypervisor_callback 1: iret xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret) .globl xen_iret_start_crit, xen_iret_end_crit /* - * This is called by xen_hypervisor_callback in entry_32.S when it sees + * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees * that the EIP at the time of interrupt was between * xen_iret_start_crit and xen_iret_end_crit. *
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret) * eflags } * cs } nested exception info * eip } - * return address : (into xen_hypervisor_callback) + * return address : (into asm_exc_xen_hypervisor_callback) * * In order to deliver the nested exception properly, we need to discard the * nested exception frame such that when we handle the exception, we do it
@@ -152,7 +152,8 @@ SYM_CODE_END(xen_iret) * * The only caveat is that if the outer eax hasn't been restored yet (i.e. * it's still on stack), we need to restore its value here. - */ +*/ +.pushsection .noinstr.text, "ax" SYM_CODE_START(xen_iret_crit_fixup) /* * Paranoia: Make sure we're really coming from kernel space.
@@ -181,3 +182,4 @@ SYM_CODE_START(xen_iret_crit_fixup) 2: ret SYM_CODE_END(xen_iret_crit_fixup) +.popsection
arch/x86/xen/xen-asm_64.S +1 -1
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_err #ifdef CONFIG_IA32_EMULATION xen_pv_trap entry_INT80_compat #endif -xen_pv_trap hypervisor_callback +xen_pv_trap asm_exc_xen_hypervisor_callback __INIT SYM_CODE_START(xen_early_idt_handler_array)
arch/x86/xen/xen-ops.h +0 -1
@@ -8,7 +8,6 @@#include <xen/xen-ops.h> /* These are code, but not functions. Defined in entry.S */ -extern const char xen_hypervisor_callback[];extern const char xen_failsafe_callback[];void xen_sysenter_target (void );
drivers/xen/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_HOTPLUG_CPU) += cpu_hotplug.o -obj-y += grant-table.o features.o balloon.o manage.o preempt.o time.o +obj-y += grant-table.o features.o balloon.o manage.o time.o obj-y += mem-reservation.o obj-y += events/ obj-y += xenbus/
drivers/xen/preempt.c deleted +0 -42
42 lines not on this page · raw message
Re: [patch V9-1 13/39] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 13 lines · Thomas Gleixner
Convert the XEN/PV hypercall to IDTENTRY:
- Emit the ASM stub with
DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Remove the old prototypesThe handler stubs need to stay in ASM code as it needs corner case handling and adjustment of the stack pointer.
Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack if required.
Quoted 8 lines
The exit code is slightly different from the regular
idtentry_exit()on non-preemptible kernels. If the hypercall is preemptible andneed_resched()is set then XEN provides a preempt hypercall scheduling function.Move this functionality into the entry code so it can use the existing idtentry functionality.
Signed-off-by Thomas Gleixner <[email protected]>
Juergen
[tip: x86/entry] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
In reply to Thomas GleixnerThe following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 66a07b44e7658da3145bb3a8f45256d103e3addc Gitweb: https://git.kernel.org/tip/66a07b44e7658da3145bb3a8f45256d103e3addc Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:26 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Convert the XEN/PV hypercall to IDTENTRY:
- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64-bit
- Remove the open coded ASM entry code in 32-bit
- Remove the old prototypes
The handler stubs need to stay in ASM code as they need corner case handling and adjustment of the stack pointer.
Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack if required.
The exit code is slightly different from the regular idtentry_exit() on non-preemptible kernels. If the hypercall is preemptible and need_resched() is set then XEN provides a preempt hypercall scheduling function.
Move this functionality into the entry code so it can use the existing idtentry functionality.
[ mingo: Build fixes. ]
--- arch/x86/entry/common.c | 74 ++++++++++++++++++++++++++++++++- arch/x86/entry/entry_32.S | 17 +++---- arch/x86/entry/entry_64.S | 20 ++------- arch/x86/include/asm/idtentry.h | 13 ++++++- arch/x86/xen/setup.c | 4 +- arch/x86/xen/smp_pv.c | 3 +- arch/x86/xen/xen-asm_32.S | 12 ++--- arch/x86/xen/xen-asm_64.S | 2 +- arch/x86/xen/xen-ops.h | 1 +- drivers/xen/Makefile | 2 +- drivers/xen/preempt.c | 42 +------------------ include/xen/xen-ops.h | 19 ++------ 12 files changed, 123 insertions(+), 86 deletions(-) delete mode 100644 drivers/xen/preempt.c
arch/x86/entry/common.c +74 -0
@@ -27,6 +27,11 @@#include <linux/syscalls.h> #include <linux/uaccess.h> +#ifdef CONFIG_XEN_PV +#include <xen/xen-ops.h> +#include <xen/events.h> +#endif +#include <asm/desc.h> #include <asm/traps.h> #include <asm/vdso.h>
@@ -35,6 +40,7 @@#include <asm/nospec-branch.h> #include <asm/io_bitmap.h> #include <asm/syscall.h> +#include <asm/irq_stack.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
@@ -680,3 +686,71 @@ void noinstr idtentry_exit_user(struct pt_regs *regs)prepare_exit_to_usermode (regs); } + +#ifdef CONFIG_XEN_PV +#ifndef CONFIG_PREEMPTION +/* +* Some hypercalls issued by the toolstack can take many 10s of +* seconds. Allow tasks running hypercalls via the privcmd driver to +* be voluntarily preempted even if full kernel preemption is +* disabled. +* +* Such preemptible hypercalls are bracketed by +* xen_preemptible_hcall_begin() and xen_preemptible_hcall_end() +* calls. +*/ +DEFINE_PER_CPU (bool, xen_in_preemptible_hcall); +EXPORT_SYMBOL_GPL (xen_in_preemptible_hcall); + +/* +* In case of scheduling the flag must be cleared and restored after +* returning from schedule as the task might move to a different CPU. +*/ +static __always_inline bool get_and_clear_inhcall(void) +{ +bool inhcall =__this_cpu_read (xen_in_preemptible_hcall); + +__this_cpu_write (xen_in_preemptible_hcall, false); +return inhcall; +} + +static __always_inline void restore_inhcall(bool inhcall) +{ +__this_cpu_write (xen_in_preemptible_hcall, inhcall); +} +#else +static __always_inline bool get_and_clear_inhcall(void) { return false; } +static __always_inline void restore_inhcall(bool inhcall) { } +#endif + +static void __xen_pv_evtchn_do_upcall (void ) +{ +irq_enter_rcu (); +inc_irq_stat (irq_hv_callback_count); + +xen_hvm_evtchn_do_upcall (); + +irq_exit_rcu (); +} + +__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +bool inhcall, rcu_exit; + + rcu_exit =idtentry_enter_cond_rcu (regs); + old_regs =set_irq_regs (regs); + +run_on_irqstack_cond (__xen_pv_evtchn_do_upcall,NULL , regs); + +set_irq_regs (old_regs); + + inhcall =get_and_clear_inhcall (); +if (inhcall && !WARN_ON_ONCE (rcu_exit)) { +idtentry_exit_cond_resched (regs, true); +restore_inhcall (inhcall); + }else { +idtentry_exit_cond_rcu (regs, rcu_exit); + } +} +#endif /* CONFIG_XEN_PV */
arch/x86/entry/entry_32.S +9 -8
@@ -1298,7 +1298,10 @@ SYM_CODE_END(native_iret) #endif #ifdef CONFIG_XEN_PV -SYM_FUNC_START(xen_hypervisor_callback) +/* + * See comment in entry_64.S for further explanation + */ +SYM_FUNC_START(exc_xen_hypervisor_callback) /* * Check to see if we got the event in the critical * region in xen_iret_direct, after we've reenabled
@@ -1315,14 +1318,11 @@ SYM_FUNC_START(xen_hypervisor_callback) pushl $-1 /* orig_ax = -1 => not a system call */ SAVE_ALL ENCODE_FRAME_POINTER - TRACE_IRQS_OFF + mov %esp, %eax - call xen_evtchn_do_upcall -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp ret_from_intr -SYM_FUNC_END(xen_hypervisor_callback) + call xen_pv_evtchn_do_upcall + jmp handle_exception_return +SYM_FUNC_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
@@ -1464,6 +1464,7 @@ SYM_CODE_START_LOCAL_NOALIGN(handle_exception) movl %esp, %eax # pt_regs pointer CALL_NOSPEC edi +handle_exception_return: #ifdef CONFIG_VM86 movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS movb PT_CS(%esp), %al
arch/x86/entry/entry_64.S +6 -14
@@ -1067,10 +1067,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1 -#ifdef CONFIG_XEN_PV -idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0 -#endif - /* * Reload gs selector with exception handling * edi: new selector
@@ -1158,9 +1154,10 @@ SYM_FUNC_END(asm_call_on_stack) * So, on entry to the handler we detect whether we interrupted an * existing activation in its critical region -- if so, we pop the current * activation and restart the handler using the previous one. + * + * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs) */ -/* do_hypervisor_callback(struct *pt_regs) */ -SYM_CODE_START_LOCAL(xen_do_hypervisor_callback) +SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) /* * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1170,15 +1167,10 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_callback) movq %rdi, %rsp /* we don't return, adjust the stack frame */ UNWIND_HINT_REGS - ENTER_IRQ_STACK old_rsp=%r10 - call xen_evtchn_do_upcall - LEAVE_IRQ_STACK + call xen_pv_evtchn_do_upcall -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp error_exit -SYM_CODE_END(xen_do_hypervisor_callback) + jmp error_return +SYM_CODE_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
arch/x86/include/asm/idtentry.h +13 -0
@@ -337,6 +337,13 @@ __visible noinstr void func(struct pt_regs *regs, \* This avoids duplicate defines and ensures that everything is consistent. */ +/* +* Dummy trap number so the low level ASM macro vector number checks do not +* match which results in emitting plain IDTENTRY stubs without bells and +* whistels. +*/ +#define X86_TRAP_OTHER 0xFFFF +/* Simple exception entry points. No hardware error code */ DECLARE_IDTENTRY (X86_TRAP_DE , exc_divide_error);DECLARE_IDTENTRY (X86_TRAP_OF , exc_overflow);
@@ -376,4 +383,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB, debug);/* #DF */ DECLARE_IDTENTRY_DF (X86_TRAP_DF , exc_double_fault); +#ifdef CONFIG_XEN_PV +DECLARE_IDTENTRY (X86_TRAP_OTHER , exc_xen_hypervisor_callback); +#endif + +#undef X86_TRAP_OTHER +#endif
arch/x86/xen/setup.c +3 -1
@@ -20,6 +20,7 @@#include <asm/setup.h> #include <asm/acpi.h> #include <asm/numa.h> +#include <asm/idtentry.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h>
@@ -993,7 +994,8 @@ static void __init xen_pvmmu_arch_setup(void)HYPERVISOR_vm_assist (VMASST_CMD_enable, VMASST_TYPE_pae_extended_cr3); -if (register_callback (CALLBACKTYPE_event, xen_hypervisor_callback) || +if (register_callback (CALLBACKTYPE_event, + xen_asm_exc_xen_hypervisor_callback) ||register_callback (CALLBACKTYPE_failsafe, xen_failsafe_callback))BUG ();
arch/x86/xen/smp_pv.c +2 -1
@@ -27,6 +27,7 @@#include <asm/paravirt.h> #include <asm/desc.h> #include <asm/pgtable.h> +#include <asm/idtentry.h> #include <asm/cpu.h> #include <xen/interface/xen.h>
@@ -347,7 +348,7 @@ cpu_initialize_context(unsigned int cpu, struct task_struct *idle) ctxt->gs_base_kernel =per_cpu_offset (cpu);#endif ctxt->event_callback_eip = - (unsigned long )xen_hypervisor_callback; + (unsigned long )xen_asm_exc_xen_hypervisor_callback; ctxt->failsafe_callback_eip = (unsigned long )xen_failsafe_callback;per_cpu (xen_cr3, cpu) =__pa (swapper_pg_dir);
arch/x86/xen/xen-asm_32.S +7 -5
@@ -93,7 +93,7 @@ xen_iret_start_crit: /* * If there's something pending, mask events again so we can - * jump back into xen_hypervisor_callback. Otherwise do not + * jump back into exc_xen_hypervisor_callback. Otherwise do not * touch XEN_vcpu_info_mask. */ jne 1f
@@ -113,7 +113,7 @@ iret_restore_end: * Events are masked, so jumping out of the critical region is * OK. */ - je xen_hypervisor_callback + je asm_exc_xen_hypervisor_callback 1: iret xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret) .globl xen_iret_start_crit, xen_iret_end_crit /* - * This is called by xen_hypervisor_callback in entry_32.S when it sees + * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees * that the EIP at the time of interrupt was between * xen_iret_start_crit and xen_iret_end_crit. *
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret) * eflags } * cs } nested exception info * eip } - * return address : (into xen_hypervisor_callback) + * return address : (into asm_exc_xen_hypervisor_callback) * * In order to deliver the nested exception properly, we need to discard the * nested exception frame such that when we handle the exception, we do it
@@ -152,7 +152,8 @@ SYM_CODE_END(xen_iret) * * The only caveat is that if the outer eax hasn't been restored yet (i.e. * it's still on stack), we need to restore its value here. - */ +*/ +.pushsection .noinstr.text, "ax" SYM_CODE_START(xen_iret_crit_fixup) /* * Paranoia: Make sure we're really coming from kernel space.
@@ -181,3 +182,4 @@ SYM_CODE_START(xen_iret_crit_fixup) 2: ret SYM_CODE_END(xen_iret_crit_fixup) +.popsection
arch/x86/xen/xen-asm_64.S +1 -1
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_error #ifdef CONFIG_IA32_EMULATION xen_pv_trap entry_INT80_compat #endif -xen_pv_trap hypervisor_callback +xen_pv_trap asm_exc_xen_hypervisor_callback __INIT SYM_CODE_START(xen_early_idt_handler_array)
arch/x86/xen/xen-ops.h +0 -1
@@ -8,7 +8,6 @@#include <xen/xen-ops.h> /* These are code, but not functions. Defined in entry.S */ -extern const char xen_hypervisor_callback[];extern const char xen_failsafe_callback[];void xen_sysenter_target (void );
drivers/xen/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_HOTPLUG_CPU) += cpu_hotplug.o -obj-y += grant-table.o features.o balloon.o manage.o preempt.o time.o +obj-y += grant-table.o features.o balloon.o manage.o time.o obj-y += mem-reservation.o obj-y += events/ obj-y += xenbus/
drivers/xen/preempt.c deleted +0 -42
42 lines not on this page · raw message
include/xen/xen-ops.h +7 -12
@@ -215,17 +215,7 @@ bool xen_running_on_version_or_later(unsigned int major, unsigned int minor);void xen_efi_runtime_setup (void ); -#ifdef CONFIG_PREEMPTION - -static inline void xen_preemptible_hcall_begin (void ) -{ -} - -static inline void xen_preemptible_hcall_end (void ) -{ -} - -#else +#if defined(CONFIG_XEN_PV ) && !defined(CONFIG_PREEMPTION )DECLARE_PER_CPU (bool, xen_in_preemptible_hcall);
@@ -239,6 +229,11 @@ static inline void xen_preemptible_hcall_end(void)__this_cpu_write (xen_in_preemptible_hcall, false); } -#endif /* CONFIG_PREEMPTION */ +#else + +static inline void xen_preemptible_hcall_begin (void ) { } +static inline void xen_preemptible_hcall_end (void ) { } + +#endif /* CONFIG_XEN_PV && !CONFIG_PREEMPTION */ #endif /* INCLUDE_XEN_OPS_H */
PATCH v9 14/39 x86/entry/64: Simplify idtentry_body
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
All C functions which do not have an error code have been converted to the new IDTENTRY interface which does not expect an error code in the arguments. Spare the XORL.
--- arch/x86/entry/entry_64.S | 2 -- 1 file changed, 2 deletions(-)
arch/x86/entry/entry_64.S +0 -2
@@ -531,8 +531,6 @@ SYM_CODE_END(spurious_entries_start) .if \has_error_code == 1 movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ - .else - xorl %esi, %esi /* Clear the error code */ .endif .if \vector == X86_TRAP_PF
[tip: x86/entry] x86/entry/64: Simplify idtentry_body
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 8518ad5d38056deee232b186533a18e7722ad3ad Gitweb: https://git.kernel.org/tip/8518ad5d38056deee232b186533a18e7722ad3ad Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:27 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry/64: Simplify idtentry_body
All C functions which do not have an error code have been converted to the new IDTENTRY interface which does not expect an error code in the arguments. Spare the XORL.
--- arch/x86/entry/entry_64.S | 2 -- 1 file changed, 2 deletions(-)
arch/x86/entry/entry_64.S +0 -2
@@ -531,8 +531,6 @@ SYM_CODE_END(spurious_entries_start) .if \has_error_code == 1 movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ - .else - xorl %esi, %esi /* Clear the error code */ .endif .if \vector == X86_TRAP_PF
PATCH v9 15/39 x86/entry: Switch page fault exception to IDTENTRY_RAW
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Convert page fault exceptions to IDTENTRY_RAW:
- Implement the C entry point with DEFINE_IDTENTRY_RAW
- Add the CR2 read into the exception handler
- Add the idtentry_enter/exit_cond_rcu() invocations in in the regular page fault handler and in the async PF part.
- Emit the ASM stub with DECLARE_IDTENTRY_RAW
- Remove the ASM idtentry in 64bit
- Remove the CR2 read from 64bit
- Remove the open coded ASM entry code in 32bit
- Fixup the XEN/PV code
- Remove the old prototypes
No functional change.
---
V9: Adopt async #PF to conditional RCU handling
--- arch/x86/entry/entry_32.S | 30 ----------------- arch/x86/entry/entry_64.S | 19 ----------- arch/x86/include/asm/idtentry.h | 3 + arch/x86/include/asm/traps.h | 11 ------ arch/x86/kernel/idt.c | 4 +- arch/x86/kernel/kvm.c | 15 +++++--- arch/x86/mm/fault.c | 69 +++++++++++++++++++++++++++------------- arch/x86/xen/enlighten_pv.c | 2 - arch/x86/xen/xen-asm_64.S | 2 - 9 files changed, 63 insertions(+), 92 deletions(-)
arch/x86/entry/entry_32.S +0 -30
@@ -1395,36 +1395,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vec #endif /* CONFIG_HYPERV */ -SYM_CODE_START(page_fault) - ASM_CLAC - pushl $do_page_fault - jmp common_exception_read_cr2 -SYM_CODE_END(page_fault) - -SYM_CODE_START_LOCAL_NOALIGN(common_exception_read_cr2) - /* the function address is in %gs's slot on the stack */ - SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1 - - ENCODE_FRAME_POINTER - - /* fixup %gs */ - GS_TO_REG %ecx - movl PT_GS(%esp), %edi - REG_TO_PTGS %ecx - SET_KERNEL_GS %ecx - - GET_CR2_INTO(%ecx) # might clobber %eax - - /* fixup orig %eax */ - movl PT_ORIG_EAX(%esp), %edx # get the error code - movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart - - TRACE_IRQS_OFF - movl %esp, %eax # pt_regs pointer - CALL_NOSPEC edi - jmp ret_from_exception -SYM_CODE_END(common_exception_read_cr2) - SYM_CODE_START_LOCAL_NOALIGN(common_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -19
@@ -506,15 +506,6 @@ SYM_CODE_END(spurious_entries_start) call error_entry UNWIND_HINT_REGS - .if \vector == X86_TRAP_PF - /* - * Store CR2 early so subsequent faults cannot clobber it. Use R12 as - * intermediate storage as RDX can be clobbered in enter_from_user_mode(). - * GET_CR2_INTO can clobber RAX. - */ - GET_CR2_INTO(%r12); - .endif - .if \sane == 0 TRACE_IRQS_OFF
@@ -533,10 +524,6 @@ SYM_CODE_END(spurious_entries_start) movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ .endif - .if \vector == X86_TRAP_PF - movq %r12, %rdx /* Move CR2 into 3rd argument */ - .endif - call \cfunc .if \sane == 0
@@ -1060,12 +1047,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work #endif /* - * Exception entry points. - */ - -idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1 - -/* * Reload gs selector with exception handling * edi: new selector *
arch/x86/include/asm/idtentry.h +2 -1
@@ -366,7 +366,8 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_GP,DECLARE_IDTENTRY_ERRORCODE (X86_TRAP_AC , exc_alignment_check);/* Raw exception entries which need extra work */ -DECLARE_IDTENTRY_RAW (X86_TRAP_BP , exc_int3); +DECLARE_IDTENTRY_RAW (X86_TRAP_BP , exc_int3); +DECLARE_IDTENTRY_RAW_ERRORCODE (X86_TRAP_PF , exc_page_fault);#ifdef CONFIG_X86_MCE DECLARE_IDTENTRY_MCE (X86_TRAP_MC , exc_machine_check);
arch/x86/include/asm/traps.h +0 -11
@@ -9,17 +9,6 @@#include <asm/idtentry.h> #include <asm/siginfo.h> /* TRAP_TRACE, ... */ -#define dotraplinkage __visible - -asmlinkage void page_fault(void); -asmlinkage void async_page_fault(void); - -#if defined(CONFIG_X86_64 ) && defined(CONFIG_XEN_PV ) -asmlinkage void xen_page_fault(void); -#endif - -dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address); -#ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs); asmlinkage __visible notrace
arch/x86/kernel/idt.c +2 -2
@@ -62,7 +62,7 @@ static const __initconst struct idt_data INTG(X86_TRAP_DB, asm_exc_debug), SYSG(X86_TRAP_BP, asm_exc_int3), #ifdef CONFIG_X86_32 - INTG(X86_TRAP_PF, page_fault), + INTG(X86_TRAP_PF, asm_exc_page_fault), #endif };
@@ -156,7 +156,7 @@ static const __initconst struct idt_data* stacks work only after cpu_init(). */ static const __initconst struct idt_data early_pf_idts[] = { -INTG (X86_TRAP_PF , page_fault), +INTG (X86_TRAP_PF , asm_exc_page_fault), };/*
arch/x86/kernel/kvm.c +9 -6
@@ -218,7 +218,7 @@ void kvm_async_pf_task_wake(u32 token) }EXPORT_SYMBOL_GPL (kvm_async_pf_task_wake); -u32 kvm_read_and_reset_pf_reason (void ) +u32 noinstr kvm_read_and_reset_pf_reason(void) {u32 reason =0 ;
@@ -230,11 +230,11 @@ u32 kvm_read_and_reset_pf_reason(void)return reason; }EXPORT_SYMBOL_GPL (kvm_read_and_reset_pf_reason); -NOKPROBE_SYMBOL (kvm_read_and_reset_pf_reason); -bool __kvm_handle_async_pf (struct pt_regs *regs,u32 token) +noinstr bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token) {u32 reason =kvm_read_and_reset_pf_reason (); +bool rcu_exit;switch (reason) {case KVM_PV_REASON_PAGE_NOT_PRESENT :
@@ -244,6 +244,9 @@ bool __kvm_handle_async_pf(struct pt_regreturn false; } + rcu_exit =idtentry_enter_cond_rcu (regs); +instrumentation_begin (); +/* * If the host managed to inject an async #PF into an interrupt * disabled region, then die hard as this is not going to end well
@@ -258,13 +261,13 @@ bool __kvm_handle_async_pf(struct pt_reg/* Page is swapped out by the host. */ kvm_async_pf_task_wait_schedule (token); }else { -rcu_irq_enter ();kvm_async_pf_task_wake (token); -rcu_irq_exit (); } + +instrumentation_end (); +idtentry_exit_cond_rcu (regs, rcu_exit);return true; } -NOKPROBE_SYMBOL (__kvm_handle_async_pf);static void __init paravirt_ops_setup (void ) {
arch/x86/mm/fault.c +48 -21
87 lines not on this page · raw message
arch/x86/xen/enlighten_pv.c +1 -1
@@ -627,7 +627,7 @@ static struct trap_array_entry trap_arra #ifdef CONFIG_IA32_EMULATION { entry_INT80_compat, xen_entry_INT80_compat, false }, #endif - { page_fault, xen_page_fault, false }, + TRAP_ENTRY(exc_page_fault, false ), TRAP_ENTRY(exc_divide_error, false ), TRAP_ENTRY(exc_bounds, false ), TRAP_ENTRY(exc_invalid_op, false ),
arch/x86/xen/xen-asm_64.S +1 -1
@@ -43,7 +43,7 @@ xen_pv_trap asm_exc_invalid_tss xen_pv_trap asm_exc_segment_not_present xen_pv_trap asm_exc_stack_segment xen_pv_trap asm_exc_general_protection -xen_pv_trap page_fault +xen_pv_trap asm_exc_page_fault xen_pv_trap asm_exc_spurious_interrupt_bug xen_pv_trap asm_exc_coprocessor_error xen_pv_trap asm_exc_alignment_check
[tip: x86/entry] x86/entry: Switch page fault exception to IDTENTRY_RAW
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: a707ae1a9bbb3f8649004d4629f4299ffdcecc2a Gitweb: https://git.kernel.org/tip/a707ae1a9bbb3f8649004d4629f4299ffdcecc2a Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:28 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Switch page fault exception to IDTENTRY_RAW
Convert page fault exceptions to IDTENTRY_RAW:
- Implement the C entry point with DEFINE_IDTENTRY_RAW
- Add the CR2 read into the exception handler
- Add the idtentry_enter/exit_cond_rcu() invocations in in the regular page fault handler and in the async PF part.
- Emit the ASM stub with DECLARE_IDTENTRY_RAW
- Remove the ASM idtentry in 64-bit
- Remove the CR2 read from 64-bit
- Remove the open coded ASM entry code in 32-bit
- Fix up the XEN/PV code
- Remove the old prototypes
No functional change.
--- arch/x86/entry/entry_32.S | 30 +-------------- arch/x86/entry/entry_64.S | 19 +--------- arch/x86/include/asm/idtentry.h | 3 +- arch/x86/include/asm/traps.h | 11 +----- arch/x86/kernel/idt.c | 4 +- arch/x86/kernel/kvm.c | 15 ++++--- arch/x86/mm/fault.c | 69 ++++++++++++++++++++++---------- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/xen-asm_64.S | 2 +- 9 files changed, 63 insertions(+), 92 deletions(-)
arch/x86/entry/entry_32.S +0 -30
@@ -1395,36 +1395,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR, #endif /* CONFIG_HYPERV */ -SYM_CODE_START(page_fault) - ASM_CLAC - pushl $do_page_fault - jmp common_exception_read_cr2 -SYM_CODE_END(page_fault) - -SYM_CODE_START_LOCAL_NOALIGN(common_exception_read_cr2) - /* the function address is in %gs's slot on the stack */ - SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1 - - ENCODE_FRAME_POINTER - - /* fixup %gs */ - GS_TO_REG %ecx - movl PT_GS(%esp), %edi - REG_TO_PTGS %ecx - SET_KERNEL_GS %ecx - - GET_CR2_INTO(%ecx) # might clobber %eax - - /* fixup orig %eax */ - movl PT_ORIG_EAX(%esp), %edx # get the error code - movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart - - TRACE_IRQS_OFF - movl %esp, %eax # pt_regs pointer - CALL_NOSPEC edi - jmp ret_from_exception -SYM_CODE_END(common_exception_read_cr2) - SYM_CODE_START_LOCAL_NOALIGN(common_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -19
@@ -506,15 +506,6 @@ SYM_CODE_END(spurious_entries_start) call error_entry UNWIND_HINT_REGS - .if \vector == X86_TRAP_PF - /* - * Store CR2 early so subsequent faults cannot clobber it. Use R12 as - * intermediate storage as RDX can be clobbered in enter_from_user_mode(). - * GET_CR2_INTO can clobber RAX. - */ - GET_CR2_INTO(%r12); - .endif - .if \sane == 0 TRACE_IRQS_OFF
@@ -533,10 +524,6 @@ SYM_CODE_END(spurious_entries_start) movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ .endif - .if \vector == X86_TRAP_PF - movq %r12, %rdx /* Move CR2 into 3rd argument */ - .endif - call \cfunc .if \sane == 0
@@ -1060,12 +1047,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt #endif /* - * Exception entry points. - */ - -idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1 - -/* * Reload gs selector with exception handling * edi: new selector *
arch/x86/include/asm/idtentry.h +2 -1
@@ -366,7 +366,8 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_GP, exc_general_protection);DECLARE_IDTENTRY_ERRORCODE (X86_TRAP_AC , exc_alignment_check);/* Raw exception entries which need extra work */ -DECLARE_IDTENTRY_RAW (X86_TRAP_BP , exc_int3); +DECLARE_IDTENTRY_RAW (X86_TRAP_BP , exc_int3); +DECLARE_IDTENTRY_RAW_ERRORCODE (X86_TRAP_PF , exc_page_fault);#ifdef CONFIG_X86_MCE DECLARE_IDTENTRY_MCE (X86_TRAP_MC , exc_machine_check);
arch/x86/include/asm/traps.h +0 -11
@@ -9,17 +9,6 @@#include <asm/idtentry.h> #include <asm/siginfo.h> /* TRAP_TRACE, ... */ -#define dotraplinkage __visible - -asmlinkage void page_fault(void); -asmlinkage void async_page_fault(void); - -#if defined(CONFIG_X86_64 ) && defined(CONFIG_XEN_PV ) -asmlinkage void xen_page_fault(void); -#endif - -dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address); -#ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs); asmlinkage __visible notrace
arch/x86/kernel/idt.c +2 -2
@@ -62,7 +62,7 @@ static const __initconst struct idt_data early_idts[] = { INTG(X86_TRAP_DB, asm_exc_debug), SYSG(X86_TRAP_BP, asm_exc_int3), #ifdef CONFIG_X86_32 - INTG(X86_TRAP_PF, page_fault), + INTG(X86_TRAP_PF, asm_exc_page_fault), #endif };
@@ -156,7 +156,7 @@ static const __initconst struct idt_data apic_idts[] = {* stacks work only after cpu_init(). */ static const __initconst struct idt_data early_pf_idts[] = { -INTG (X86_TRAP_PF , page_fault), +INTG (X86_TRAP_PF , asm_exc_page_fault), };/*
arch/x86/kernel/kvm.c +9 -6
@@ -218,7 +218,7 @@ again: }EXPORT_SYMBOL_GPL (kvm_async_pf_task_wake); -u32 kvm_read_and_reset_pf_reason (void ) +u32 noinstr kvm_read_and_reset_pf_reason(void) {u32 reason =0 ;
@@ -230,11 +230,11 @@ u32 kvm_read_and_reset_pf_reason(void)return reason; }EXPORT_SYMBOL_GPL (kvm_read_and_reset_pf_reason); -NOKPROBE_SYMBOL (kvm_read_and_reset_pf_reason); -bool __kvm_handle_async_pf (struct pt_regs *regs,u32 token) +noinstr bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token) {u32 reason =kvm_read_and_reset_pf_reason (); +bool rcu_exit;switch (reason) {case KVM_PV_REASON_PAGE_NOT_PRESENT :
@@ -244,6 +244,9 @@ bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)return false; } + rcu_exit =idtentry_enter_cond_rcu (regs); +instrumentation_begin (); +/* * If the host managed to inject an async #PF into an interrupt * disabled region, then die hard as this is not going to end well
@@ -258,13 +261,13 @@ bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)/* Page is swapped out by the host. */ kvm_async_pf_task_wait_schedule (token); }else { -rcu_irq_enter ();kvm_async_pf_task_wake (token); -rcu_irq_exit (); } + +instrumentation_end (); +idtentry_exit_cond_rcu (regs, rcu_exit);return true; } -NOKPROBE_SYMBOL (__kvm_handle_async_pf);static void __init paravirt_ops_setup (void ) {
arch/x86/mm/fault.c +48 -21
87 lines not on this page · raw message
arch/x86/xen/enlighten_pv.c +1 -1
@@ -627,7 +627,7 @@ static struct trap_array_entry trap_array[] = { #ifdef CONFIG_IA32_EMULATION { entry_INT80_compat, xen_entry_INT80_compat, false }, #endif - { page_fault, xen_page_fault, false }, + TRAP_ENTRY(exc_page_fault, false ), TRAP_ENTRY(exc_divide_error, false ), TRAP_ENTRY(exc_bounds, false ), TRAP_ENTRY(exc_invalid_op, false ),
arch/x86/xen/xen-asm_64.S +1 -1
@@ -43,7 +43,7 @@ xen_pv_trap asm_exc_invalid_tss xen_pv_trap asm_exc_segment_not_present xen_pv_trap asm_exc_stack_segment xen_pv_trap asm_exc_general_protection -xen_pv_trap page_fault +xen_pv_trap asm_exc_page_fault xen_pv_trap asm_exc_spurious_interrupt_bug xen_pv_trap asm_exc_coprocessor_error xen_pv_trap asm_exc_alignment_check
PATCH v9 16/39 x86/entry: Remove the transition leftovers
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Now that all exceptions are converted over the sane flag is not longer needed. Also the vector argument of idtentry_body on 64 bit is pointless now.
--- arch/x86/entry/entry_32.S | 3 +-- arch/x86/entry/entry_64.S | 26 ++++---------------------- arch/x86/include/asm/idtentry.h | 6 +++--- 3 files changed, 8 insertions(+), 27 deletions(-)
arch/x86/entry/entry_32.S +1 -2
@@ -734,9 +734,8 @@ * @asmsym: ASM symbol for the entry point * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Compatibility flag with 64bit */ -.macro idtentry vector asmsym cfunc has_error_code:req sane=0 +.macro idtentry vector asmsym cfunc has_error_code:req SYM_CODE_START(\asmsym) ASM_CLAC cld
arch/x86/entry/entry_64.S +4 -22
@@ -496,27 +496,14 @@ SYM_CODE_END(spurious_entries_start) /** * idtentry_body - Macro to emit code calling the C function - * @vector: Vector number * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Sane variant which handles irq tracing, context tracking in C */ -.macro idtentry_body vector cfunc has_error_code:req sane=0 +.macro idtentry_body cfunc has_error_code:req call error_entry UNWIND_HINT_REGS - .if \sane == 0 - TRACE_IRQS_OFF - -#ifdef CONFIG_CONTEXT_TRACKING - testb $3, CS(%rsp) - jz .Lfrom_kernel_no_ctxt_tracking_\@ - CALL_enter_from_user_mode -.Lfrom_kernel_no_ctxt_tracking_\@: -#endif - .endif - movq %rsp, %rdi /* pt_regs pointer into 1st argument*/ .if \has_error_code == 1
@@ -526,11 +513,7 @@ SYM_CODE_END(spurious_entries_start) call \cfunc - .if \sane == 0 - jmp error_exit - .else jmp error_return - .endif .endm /**
@@ -539,12 +522,11 @@ SYM_CODE_END(spurious_entries_start) * @asmsym: ASM symbol for the entry point * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Sane variant which handles irq tracing, context tracking in C * * The macro emits code to set up the kernel context for straight forward * and simple IDT entries. No IST stack, no paranoid entry checks. */ -.macro idtentry vector asmsym cfunc has_error_code:req sane=0 +.macro idtentry vector asmsym cfunc has_error_code:req SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8 ASM_CLAC
@@ -567,7 +549,7 @@ SYM_CODE_START(\asmsym) .Lfrom_usermode_no_gap_\@: .endif - idtentry_body \vector \cfunc \has_error_code \sane + idtentry_body \cfunc \has_error_code _ASM_NOKPROBE(\asmsym) SYM_CODE_END(\asmsym)
@@ -642,7 +624,7 @@ SYM_CODE_START(\asmsym) /* Switch to the regular task stack and use the noist entry point */ .Lfrom_usermode_switch_stack_\@: - idtentry_body vector noist_\cfunc, has_error_code=0 sane=1 + idtentry_body noist_\cfunc, has_error_code=0 _ASM_NOKPROBE(\asmsym) SYM_CODE_END(\asmsym)
arch/x86/include/asm/idtentry.h +3 -3
@@ -283,10 +283,10 @@ static __always_inline void __##func(str* The ASM variants for DECLARE_IDTENTRY*() which emit the ASM entry stubs. */ #define DECLARE_IDTENTRY (vector, func) \ - idtentry vector asm_##func func has_error_code=0 sane=1 + idtentry vector asm_##func func has_error_code=0#define DECLARE_IDTENTRY_ERRORCODE (vector, func) \ - idtentry vector asm_##func func has_error_code=1 sane=1 + idtentry vector asm_##func func has_error_code=1/* Special case for 32bit IRET 'trap'. Do not emit ASM code */ #define DECLARE_IDTENTRY_SW (vector, func)
@@ -324,7 +324,7 @@ static __always_inline void __##func(str/* XEN NMI and DB wrapper */ #define DECLARE_IDTENTRY_XEN (vector, func) \ - idtentry vector asm_exc_xen##func exc_##func has_error_code=0 sane=1 + idtentry vector asm_exc_xen##func exc_##func has_error_code=0#endif /* __ASSEMBLY__ */
[tip: x86/entry] x86/entry: Remove the transition leftovers
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: ae8f8f2116d86618cf43be142c495906a17811ef Gitweb: https://git.kernel.org/tip/ae8f8f2116d86618cf43be142c495906a17811ef Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:29 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Remove the transition leftovers
Now that all exceptions are converted over the sane flag is not longer needed. Also the vector argument of idtentry_body on 64-bit is pointless now.
--- arch/x86/entry/entry_32.S | 3 +-- arch/x86/entry/entry_64.S | 26 ++++---------------------- arch/x86/include/asm/idtentry.h | 6 +++--- 3 files changed, 8 insertions(+), 27 deletions(-)
arch/x86/entry/entry_32.S +1 -2
@@ -734,9 +734,8 @@ * @asmsym: ASM symbol for the entry point * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Compatibility flag with 64bit */ -.macro idtentry vector asmsym cfunc has_error_code:req sane=0 +.macro idtentry vector asmsym cfunc has_error_code:req SYM_CODE_START(\asmsym) ASM_CLAC cld
arch/x86/entry/entry_64.S +4 -22
@@ -496,27 +496,14 @@ SYM_CODE_END(spurious_entries_start) /** * idtentry_body - Macro to emit code calling the C function - * @vector: Vector number * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Sane variant which handles irq tracing, context tracking in C */ -.macro idtentry_body vector cfunc has_error_code:req sane=0 +.macro idtentry_body cfunc has_error_code:req call error_entry UNWIND_HINT_REGS - .if \sane == 0 - TRACE_IRQS_OFF - -#ifdef CONFIG_CONTEXT_TRACKING - testb $3, CS(%rsp) - jz .Lfrom_kernel_no_ctxt_tracking_\@ - CALL_enter_from_user_mode -.Lfrom_kernel_no_ctxt_tracking_\@: -#endif - .endif - movq %rsp, %rdi /* pt_regs pointer into 1st argument*/ .if \has_error_code == 1
@@ -526,11 +513,7 @@ SYM_CODE_END(spurious_entries_start) call \cfunc - .if \sane == 0 - jmp error_exit - .else jmp error_return - .endif .endm /**
@@ -539,12 +522,11 @@ SYM_CODE_END(spurious_entries_start) * @asmsym: ASM symbol for the entry point * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Sane variant which handles irq tracing, context tracking in C * * The macro emits code to set up the kernel context for straight forward * and simple IDT entries. No IST stack, no paranoid entry checks. */ -.macro idtentry vector asmsym cfunc has_error_code:req sane=0 +.macro idtentry vector asmsym cfunc has_error_code:req SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8 ASM_CLAC
@@ -567,7 +549,7 @@ SYM_CODE_START(\asmsym) .Lfrom_usermode_no_gap_\@: .endif - idtentry_body \vector \cfunc \has_error_code \sane + idtentry_body \cfunc \has_error_code _ASM_NOKPROBE(\asmsym) SYM_CODE_END(\asmsym)
@@ -642,7 +624,7 @@ SYM_CODE_START(\asmsym) /* Switch to the regular task stack and use the noist entry point */ .Lfrom_usermode_switch_stack_\@: - idtentry_body vector noist_\cfunc, has_error_code=0 sane=1 + idtentry_body noist_\cfunc, has_error_code=0 _ASM_NOKPROBE(\asmsym) SYM_CODE_END(\asmsym)
arch/x86/include/asm/idtentry.h +3 -3
@@ -283,10 +283,10 @@ __visible noinstr void func(struct pt_regs *regs, \* The ASM variants for DECLARE_IDTENTRY*() which emit the ASM entry stubs. */ #define DECLARE_IDTENTRY (vector, func) \ - idtentry vector asm_##func func has_error_code=0 sane=1 + idtentry vector asm_##func func has_error_code=0#define DECLARE_IDTENTRY_ERRORCODE (vector, func) \ - idtentry vector asm_##func func has_error_code=1 sane=1 + idtentry vector asm_##func func has_error_code=1/* Special case for 32bit IRET 'trap'. Do not emit ASM code */ #define DECLARE_IDTENTRY_SW (vector, func)
@@ -324,7 +324,7 @@ __visible noinstr void func(struct pt_regs *regs, \/* XEN NMI and DB wrapper */ #define DECLARE_IDTENTRY_XEN (vector, func) \ - idtentry vector asm_exc_xen##func exc_##func has_error_code=0 sane=1 + idtentry vector asm_exc_xen##func exc_##func has_error_code=0#endif /* __ASSEMBLY__ */
PATCH v9 17/39 x86/entry: Change exit path of xen_failsafe_callback
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
xen_failsafe_callback is invoked from XEN for two cases:
1. Fault while reloading DS, ES, FS or GS
2. Fault while executing IRET
#1 retries the IRET after XEN has fixed up the segments. #2 injects a #GP which kills the task
For #1 there is no reason to go through the full exception return path because the tasks TIF state is still the same. So just going straight to the IRET path is good enough.
--- arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/entry_64.S | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/entry/entry_32.S +1 -1
@@ -1352,7 +1352,7 @@ SYM_FUNC_START(xen_failsafe_callback) 5: pushl $-1 /* orig_ax = -1 => not a system call */ SAVE_ALL ENCODE_FRAME_POINTER - jmp ret_from_exception + jmp handle_exception_return .section .fixup, "ax" 6: xorl %eax, %eax
arch/x86/entry/entry_64.S +1 -1
@@ -1175,7 +1175,7 @@ SYM_CODE_START(xen_failsafe_callback) pushq $-1 /* orig_ax = -1 => not a system call */ PUSH_AND_CLEAR_REGS ENCODE_FRAME_POINTER - jmp error_exit + jmp error_return SYM_CODE_END(xen_failsafe_callback) #endif /* CONFIG_XEN_PV */
[tip: x86/entry] x86/entry: Change exit path of xen_failsafe_callback
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: c8af31cbb0650b2be9b6a2a9cdb3a2a0770d7417 Gitweb: https://git.kernel.org/tip/c8af31cbb0650b2be9b6a2a9cdb3a2a0770d7417 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:30 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Change exit path of xen_failsafe_callback
xen_failsafe_callback() is invoked from XEN for two cases:
1. Fault while reloading DS, ES, FS or GS
2. Fault while executing IRET
#1 retries the IRET after XEN has fixed up the segments. #2 injects a #GP which kills the task
For #1 there is no reason to go through the full exception return path because the tasks TIF state is still the same. So just going straight to the IRET path is good enough.
--- arch/x86/entry/entry_32.S | 2 +- arch/x86/entry/entry_64.S | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/entry/entry_32.S +1 -1
@@ -1352,7 +1352,7 @@ SYM_FUNC_START(xen_failsafe_callback) 5: pushl $-1 /* orig_ax = -1 => not a system call */ SAVE_ALL ENCODE_FRAME_POINTER - jmp ret_from_exception + jmp handle_exception_return .section .fixup, "ax" 6: xorl %eax, %eax
arch/x86/entry/entry_64.S +1 -1
@@ -1175,7 +1175,7 @@ SYM_CODE_START(xen_failsafe_callback) pushq $-1 /* orig_ax = -1 => not a system call */ PUSH_AND_CLEAR_REGS ENCODE_FRAME_POINTER - jmp error_exit + jmp error_return SYM_CODE_END(xen_failsafe_callback) #endif /* CONFIG_XEN_PV */
PATCH v9 18/39 x86/entry/64: Remove error_exit
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
No more users.
--- arch/x86/entry/entry_64.S | 9 --------- 1 file changed, 9 deletions(-)
arch/x86/entry/entry_64.S +0 -9
@@ -1356,15 +1356,6 @@ SYM_CODE_START_LOCAL(error_entry) jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry) -SYM_CODE_START_LOCAL(error_exit) - UNWIND_HINT_REGS - DISABLE_INTERRUPTS(CLBR_ANY) - TRACE_IRQS_OFF - testb $3, CS(%rsp) - jz retint_kernel - jmp .Lretint_user -SYM_CODE_END(error_exit) - SYM_CODE_START_LOCAL(error_return) UNWIND_HINT_REGS DEBUG_ENTRY_ASSERT_IRQS_OFF
[tip: x86/entry] x86/entry/64: Remove error_exit()
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 88d41f530736bdbed258df357e97956e8f97856d Gitweb: https://git.kernel.org/tip/88d41f530736bdbed258df357e97956e8f97856d Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:31 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry/64: Remove error_exit()
No more users.
--- arch/x86/entry/entry_64.S | 9 --------- 1 file changed, 9 deletions(-)
arch/x86/entry/entry_64.S +0 -9
@@ -1356,15 +1356,6 @@ SYM_CODE_START_LOCAL(error_entry) jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry) -SYM_CODE_START_LOCAL(error_exit) - UNWIND_HINT_REGS - DISABLE_INTERRUPTS(CLBR_ANY) - TRACE_IRQS_OFF - testb $3, CS(%rsp) - jz retint_kernel - jmp .Lretint_user -SYM_CODE_END(error_exit) - SYM_CODE_START_LOCAL(error_return) UNWIND_HINT_REGS DEBUG_ENTRY_ASSERT_IRQS_OFF
PATCH v9 19/39 x86/entry/32: Remove common_exception
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
No more users.
--- arch/x86/entry/entry_32.S | 21 --------------------- 1 file changed, 21 deletions(-)
arch/x86/entry/entry_32.S +0 -21
@@ -1394,27 +1394,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vec #endif /* CONFIG_HYPERV */ -SYM_CODE_START_LOCAL_NOALIGN(common_exception) - /* the function address is in %gs's slot on the stack */ - SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1 - ENCODE_FRAME_POINTER - - /* fixup %gs */ - GS_TO_REG %ecx - movl PT_GS(%esp), %edi # get the function address - REG_TO_PTGS %ecx - SET_KERNEL_GS %ecx - - /* fixup orig %eax */ - movl PT_ORIG_EAX(%esp), %edx # get the error code - movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart - - TRACE_IRQS_OFF - movl %esp, %eax # pt_regs pointer - CALL_NOSPEC edi - jmp ret_from_exception -SYM_CODE_END(common_exception) - SYM_CODE_START_LOCAL_NOALIGN(handle_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
[tip: x86/entry] x86/entry/32: Remove common_exception()
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: b890361ba0f4a8eb565f2d8c6e045a70abc365eb Gitweb: https://git.kernel.org/tip/b890361ba0f4a8eb565f2d8c6e045a70abc365eb Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:32 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry/32: Remove common_exception()
No more users.
--- arch/x86/entry/entry_32.S | 21 --------------------- 1 file changed, 21 deletions(-)
arch/x86/entry/entry_32.S +0 -21
@@ -1394,27 +1394,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR, #endif /* CONFIG_HYPERV */ -SYM_CODE_START_LOCAL_NOALIGN(common_exception) - /* the function address is in %gs's slot on the stack */ - SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1 - ENCODE_FRAME_POINTER - - /* fixup %gs */ - GS_TO_REG %ecx - movl PT_GS(%esp), %edi # get the function address - REG_TO_PTGS %ecx - SET_KERNEL_GS %ecx - - /* fixup orig %eax */ - movl PT_ORIG_EAX(%esp), %edx # get the error code - movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart - - TRACE_IRQS_OFF - movl %esp, %eax # pt_regs pointer - CALL_NOSPEC edi - jmp ret_from_exception -SYM_CODE_END(common_exception) - SYM_CODE_START_LOCAL_NOALIGN(handle_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
PATCH v9 20/39 x86/irq: Use generic irq_regs implementation
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
The only difference is the name of the per-CPU variable: irq_regs vs. __irq_regs, but the accessor functions are identical.
Remove the pointless copy and use the generic variant.
--- arch/x86/include/asm/irq_regs.h | 32 -------------------------------- arch/x86/kernel/irq.c | 3 --- 2 files changed, 35 deletions(-)
arch/x86/include/asm/irq_regs.h deleted +0 -32
@@ -1,32 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* -* Per-cpu current frame pointer - the location of the last exception frame on -* the stack, stored in the per-cpu area. -* -* Jeremy Fitzhardinge <[email protected]> -*/ -#ifndef _ASM_X86_IRQ_REGS_H -#define _ASM_X86_IRQ_REGS_H - -#include <asm/percpu.h> - -#define ARCH_HAS_OWN_IRQ_REGS - -DECLARE_PER_CPU(struct pt_regs *, irq_regs); - -static inline struct pt_regs *get_irq_regs (void ) -{ -return __this_cpu_read (irq_regs); -} - -static inline struct pt_regs *set_irq_regs (struct pt_regs *new_regs) -{ -struct pt_regs *old_regs; - - old_regs =get_irq_regs (); -__this_cpu_write (irq_regs, new_regs); - -return old_regs; -} - -#endif /* _ASM_X86_IRQ_REGS_32_H */
arch/x86/kernel/irq.c +0 -3
@@ -26,9 +26,6 @@DEFINE_PER_CPU_SHARED_ALIGNED (irq_cpustat_t, irq_stat);EXPORT_PER_CPU_SYMBOL (irq_stat); -DEFINE_PER_CPU(struct pt_regs *, irq_regs); -EXPORT_PER_CPU_SYMBOL (irq_regs); -atomic_t irq_err_count;/*
Re: [patch V9 20/39] x86/irq: Use generic irq_regs implementation
Hello Thomas,
with this patch i got the following build error:
LD .tmp_vmlinux.kallsyms1/home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function `get_irq_regs':/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:19: undefined reference to irq_regs' /home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function set_irq_regs':/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:27: undefined reference to `irq_regs' /home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: /home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:27: undefined reference to irq_regs' /home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function get_irq_regs':/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:19: undefined reference to irq_regs' /home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function set_irq_regs':/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:27: undefined reference to `irq_regs'
Best regards
Damian
Re: [patch V9 20/39] x86/irq: Use generic irq_regs implementation
Damian,
Quoted 5 lines · damian
with this patch i got the following build error:
LD .tmp_vmlinux.kallsyms1
/home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld:arch/x86/kernel/irq.o: in function `get_irq_regs':/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:19: undefined reference to `irq_regs'
Can you provide the .config please?
Thanks,
tglx
Re: [patch V9 20/39] x86/irq: Use generic irq_regs implementation
Quoted 9 lines · Thomas Gleixner
Damian,
Quoted 5 lines · damian
with this patch i got the following build error:
LD .tmp_vmlinux.kallsyms1
/home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld:arch/x86/kernel/irq.o: in function `get_irq_regs':/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:19: undefined reference to `irq_regs'
Quoted 5 lines
Can you provide the .config please?
Thanks,
tglx
Hello Thomas,
I found out self the reason. I rebuild the kernel from scratch again and now it works.
Many Thnaks
Damian
[tip: x86/entry] x86/irq: Use generic irq_regs implementation
In reply to Thomas GleixnerThe following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 359763a015a4bc3ac9899bf8a3ea97dae91c8e2a Gitweb: https://git.kernel.org/tip/359763a015a4bc3ac9899bf8a3ea97dae91c8e2a Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:33 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/irq: Use generic irq_regs implementation
The only difference is the name of the per-CPU variable: irq_regs vs. __irq_regs, but the accessor functions are identical.
Remove the pointless copy and use the generic variant.
--- arch/x86/include/asm/irq_regs.h | 32 +-------------------------------- arch/x86/kernel/irq.c | 3 +--- 2 files changed, 35 deletions(-) delete mode 100644 arch/x86/include/asm/irq_regs.h
arch/x86/include/asm/irq_regs.h deleted +0 -32
@@ -1,32 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* -* Per-cpu current frame pointer - the location of the last exception frame on -* the stack, stored in the per-cpu area. -* -* Jeremy Fitzhardinge <[email protected]> -*/ -#ifndef _ASM_X86_IRQ_REGS_H -#define _ASM_X86_IRQ_REGS_H - -#include <asm/percpu.h> - -#define ARCH_HAS_OWN_IRQ_REGS - -DECLARE_PER_CPU(struct pt_regs *, irq_regs); - -static inline struct pt_regs *get_irq_regs (void ) -{ -return __this_cpu_read (irq_regs); -} - -static inline struct pt_regs *set_irq_regs (struct pt_regs *new_regs) -{ -struct pt_regs *old_regs; - - old_regs =get_irq_regs (); -__this_cpu_write (irq_regs, new_regs); - -return old_regs; -} - -#endif /* _ASM_X86_IRQ_REGS_32_H */
arch/x86/kernel/irq.c +0 -3
@@ -26,9 +26,6 @@DEFINE_PER_CPU_SHARED_ALIGNED (irq_cpustat_t, irq_stat);EXPORT_PER_CPU_SYMBOL (irq_stat); -DEFINE_PER_CPU(struct pt_regs *, irq_regs); -EXPORT_PER_CPU_SYMBOL (irq_regs); -atomic_t irq_err_count;/*
PATCH v9 21/39 x86/irq: Convey vector as argument and not in ptregs
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Device interrupts which go through do_IRQ() or the spurious interrupt handler have their separate entry code on 64 bit for no good reason.
Both 32 and 64 bit transport the vector number through ORIG_[RE]AX in pt_regs. Further the vector number is forced to fit into an u8 and is complemented and offset by 0x80 so it's in the signed character range. Otherwise GAS would expand the pushq to a 5 byte instruction for any vector > 0x7F.
Treat the vector number like an error code and hand it to the C function as argument. This allows to get rid of the extra entry code in a later step.
Simplify the error code push magic by implementing the pushq imm8 via a '.byte 0x6a, vector' sequence so GAS is not able to screw it up. As the pushq imm8 is sign extending the resulting error code needs to be truncated to 8 bits in C code.
---
V9: Add protection against GAS emitting more than 8 byte per vector
--- arch/x86/entry/calling.h | 5 +++- arch/x86/entry/entry_32.S | 33 +++------------------------- arch/x86/entry/entry_64.S | 40 ++++++---------------------------- arch/x86/include/asm/entry_arch.h | 2 - arch/x86/include/asm/hw_irq.h | 1 arch/x86/include/asm/idtentry.h | 44 ++++++++++++++++++++++++++++++++++++++ arch/x86/include/asm/irq.h | 2 - arch/x86/include/asm/traps.h | 3 +- arch/x86/kernel/apic/apic.c | 31 +++++++++++++++++++++----- arch/x86/kernel/idt.c | 2 - arch/x86/kernel/irq.c | 14 +++++++----- 11 files changed, 99 insertions(+), 78 deletions(-)
arch/x86/entry/calling.h +4 -1
@@ -341,7 +341,10 @@ For 32-bit we have the following convent#endif .endm -#endif /* CONFIG_X86_64 */ +#else /* CONFIG_X86_64 */ +# undef UNWIND_HINT_IRET_REGS +# define UNWIND_HINT_IRET_REGS +#endif /* !CONFIG_X86_64 */ .macro STACKLEAK_ERASE#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
arch/x86/entry/entry_32.S +4 -29
@@ -1215,40 +1215,15 @@ SYM_FUNC_END(entry_INT80_32) #endif .endm -/* - * Build the entry stubs with some assembler magic. - * We pack 1 stub into every 8-byte block. - */ - .align 8 -SYM_CODE_START(irq_entries_start) - vector=FIRST_EXTERNAL_VECTOR - .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) - pushl $(~vector+0x80) /* Note: always in signed byte range */ - vector=vector+1 - jmp common_interrupt - .align 8 - .endr -SYM_CODE_END(irq_entries_start) - #ifdef CONFIG_X86_LOCAL_APIC - .align 8 -SYM_CODE_START(spurious_entries_start) - vector=FIRST_SYSTEM_VECTOR - .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) - pushl $(~vector+0x80) /* Note: always in signed byte range */ - vector=vector+1 - jmp common_spurious - .align 8 - .endr -SYM_CODE_END(spurious_entries_start) - SYM_CODE_START_LOCAL(common_spurious) ASM_CLAC - addl $-0x80, (%esp) /* Adjust vector into the [-256, -1] range */ SAVE_ALL switch_stacks=1 ENCODE_FRAME_POINTER TRACE_IRQS_OFF movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ call smp_spurious_interrupt jmp ret_from_intr SYM_CODE_END(common_spurious)
@@ -1261,12 +1236,12 @@ SYM_CODE_END(common_spurious) .p2align CONFIG_X86_L1_CACHE_SHIFT SYM_CODE_START_LOCAL(common_interrupt) ASM_CLAC - addl $-0x80, (%esp) /* Adjust vector into the [-256, -1] range */ - SAVE_ALL switch_stacks=1 ENCODE_FRAME_POINTER TRACE_IRQS_OFF movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ call do_IRQ jmp ret_from_intr SYM_CODE_END(common_interrupt)
arch/x86/entry/entry_64.S +7 -33
@@ -358,34 +358,6 @@ SYM_CODE_START(ret_from_fork) SYM_CODE_END(ret_from_fork) .popsection -/* - * Build the entry stubs with some assembler magic. - * We pack 1 stub into every 8-byte block. - */ - .align 8 -SYM_CODE_START(irq_entries_start) - vector=FIRST_EXTERNAL_VECTOR - .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) - UNWIND_HINT_IRET_REGS - pushq $(~vector+0x80) /* Note: always in signed byte range */ - jmp common_interrupt - .align 8 - vector=vector+1 - .endr -SYM_CODE_END(irq_entries_start) - - .align 8 -SYM_CODE_START(spurious_entries_start) - vector=FIRST_SYSTEM_VECTOR - .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) - UNWIND_HINT_IRET_REGS - pushq $(~vector+0x80) /* Note: always in signed byte range */ - jmp common_spurious - .align 8 - vector=vector+1 - .endr -SYM_CODE_END(spurious_entries_start) - .macro DEBUG_ENTRY_ASSERT_IRQS_OFF #ifdef CONFIG_DEBUG_ENTRY pushq %rax
@@ -755,13 +727,14 @@ SYM_CODE_END(interrupt_entry) /* Interrupt entry/exit. */ /* - * The interrupt stubs push (~vector+0x80) onto the stack and + * The interrupt stubs push vector onto the stack and * then jump to common_spurious/interrupt. */ SYM_CODE_START_LOCAL(common_spurious) - addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1 + movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ + movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ call smp_spurious_interrupt /* rdi points to pt_regs */ jmp ret_from_intr SYM_CODE_END(common_spurious)
@@ -770,10 +743,11 @@ SYM_CODE_END(common_spurious) /* common_interrupt is a hotpath. Align it */ .p2align CONFIG_X86_L1_CACHE_SHIFT SYM_CODE_START_LOCAL(common_interrupt) - addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1 - call do_IRQ /* rdi points to pt_regs */ + movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ + movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ + call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY)
@@ -1022,7 +996,7 @@ apicinterrupt RESCHEDULE_VECTOR resche #endif apicinterrupt ERROR_APIC_VECTOR error_interrupt smp_error_interrupt -apicinterrupt SPURIOUS_APIC_VECTOR spurious_interrupt smp_spurious_interrupt +apicinterrupt SPURIOUS_APIC_VECTOR spurious_apic_interrupt smp_spurious_apic_interrupt #ifdef CONFIG_IRQ_WORK apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt
arch/x86/include/asm/entry_arch.h +1 -1
@@ -35,7 +35,7 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_i BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR) BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR) -BUILD_INTERRUPT(spurious_interrupt,SPURIOUS_APIC_VECTOR) +BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR) BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR)#ifdef CONFIG_IRQ_WORK
arch/x86/include/asm/hw_irq.h +1 -0
@@ -39,6 +39,7 @@ extern asmlinkage void irq_work_interrup extern asmlinkage void uv_bau_message_intr1(void); extern asmlinkage void spurious_interrupt(void); +extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +44 -0
50 lines not on this page · raw message
arch/x86/include/asm/irq.h +1 -1
@@ -36,7 +36,7 @@ extern void native_init_IRQ(void);extern void handle_irq (struct irq_desc *desc,struct pt_regs *regs); -extern __visible void do_IRQ(struct pt_regs *regs); +extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);extern void init_ISA_irqs (void );
arch/x86/include/asm/traps.h +2 -1
@@ -41,8 +41,9 @@ asmlinkage void smp_deferred_error_inter#endif void smp_apic_timer_interrupt (struct pt_regs *regs); -void smp_spurious_interrupt (struct pt_regs *regs);void smp_error_interrupt (struct pt_regs *regs); +void smp_spurious_apic_interrupt (struct pt_regs *regs); +void smp_spurious_interrupt (struct pt_regs *regs,unsigned long vector); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +25 -6
@@ -2153,15 +2153,29 @@ void __init register_lapic_address(unsig* Local APIC interrupts */ -/* -* This interrupt should _never_ happen with our APIC/SMP architecture +/** +* smp_spurious_interrupt - Catch all for interrupts raised on unused vectors +* @regs: Pointer to pt_regs on stack +* @error_code: The vector number is in the lower 8 bits +* +* This is invoked from ASM entry code to catch all interrupts which +* trigger on an entry which is routed to the common_spurious idtentry +* point. +* +* Also called from smp_spurious_apic_interrupt(). */ -__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs) +__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs, +unsigned long vector) { -u8 vector = ~regs->orig_ax;u32 v;entering_irq (); +/* +* The push in the entry ASM code which stores the vector number on +* the stack in the error code slot is sign expanding. Just use the +* lower 8 bits. +*/ + vector &=0xFF ;trace_spurious_apic_entry (vector);inc_irq_stat (irq_spurious_count);
@@ -2182,11 +2196,11 @@ void __init register_lapic_address(unsig*/ v =apic_read (APIC_ISR + ((vector & ~0x1f ) >>1 ));if (v & (1 << (vector &0x1f ))) { -pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n" , +pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n" , vector,smp_processor_id ());ack_APIC_irq (); }else { -pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n" , +pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n" , vector,smp_processor_id ()); } out:
@@ -2194,6 +2208,11 @@ void __init register_lapic_address(unsigexiting_irq (); } +__visible void smp_spurious_apic_interrupt(struct pt_regs *regs) +{ +smp_spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +} +/* * This interrupt should never happen with our APIC/SMP architecture */
arch/x86/kernel/idt.c +1 -1
@@ -145,7 +145,7 @@ static const __initconst struct idt_data #ifdef CONFIG_X86_UV INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), #endif - INTG(SPURIOUS_APIC_VECTOR, spurious_interrupt), + INTG(SPURIOUS_APIC_VECTOR, spurious_apic_interrupt), INTG(ERROR_APIC_VECTOR, error_interrupt), #endif };
arch/x86/kernel/irq.c +9 -5
@@ -227,14 +227,18 @@ u64 arch_irq_stat(void)* SMP cross-CPU interrupts have their own specific * handlers). */ -__visible void __irq_entry do_IRQ(struct pt_regs *regs) +__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) {struct pt_regs *old_regs =set_irq_regs (regs); -struct irq_desc * desc; -/* high bit used in ret_from_ code */ -unsigned vector = ~regs->orig_ax; +struct irq_desc *desc;entering_irq (); +/* +* The push in the entry ASM code which stores the vector number on +* the stack in the error code slot is sign expanding. Just use the +* lower 8 bits. +*/ + vector &=0xFF ;/* entering_irq() tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" );
@@ -249,7 +253,7 @@ u64 arch_irq_stat(void)ack_APIC_irq ();if (desc ==VECTOR_UNUSED ) { -pr_emerg_ratelimited ("%s: %d.%d No irq handler for vector\n" , +pr_emerg_ratelimited ("%s: %d.%lu No irq handler for vector\n" , __func__,smp_processor_id (), vector); }else {
Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
Review
Quoted 9 lines · Thomas Gleixner
(quoted diff) +9
-0+ .align 8 +SYM_CODE_START(irq_entries_start) + vector=FIRST_EXTERNAL_VECTOR + pos = . + .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) + UNWIND_HINT_IRET_REGS + .byte 0x6a, vector + jmp common_interrupt + .nops (pos + 8) - .
Boris pointed out that .nops requires binutils >=2.31 ... Sigh!
The below delta patch works for the purpose.
Thanks,
tglx
arch/x86/include/asm/idtentry.h +6 -2
@@ -475,7 +475,9 @@ SYM_CODE_START(irq_entries_start)UNWIND_HINT_IRET_REGS .byte 0x6a, vector jmp asm_common_interrupt - .nops (pos + 8) - . + nop +/* Ensure that the above is 8 bytes max */ + . = pos + 8 pos=pos+8 vector=vector+1 .endr
@@ -490,7 +492,9 @@ SYM_CODE_START(spurious_entries_start)UNWIND_HINT_IRET_REGS .byte 0x6a, vector jmp asm_spurious_interrupt - .nops (pos + 8) - . + nop +/* Ensure that the above is 8 bytes max */ + . = pos + 8 pos=pos+8 vector=vector+1 .endr
[tip: x86/entry] x86/irq: Convey vector as argument and not in ptregs
In reply to Thomas GleixnerThe following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 2f83c60d35e049112189fb9e30c346c1b97c70e4 Gitweb: https://git.kernel.org/tip/2f83c60d35e049112189fb9e30c346c1b97c70e4 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:34 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/irq: Convey vector as argument and not in ptregs
Device interrupts which go through do_IRQ() or the spurious interrupt handler have their separate entry code on 64 bit for no good reason.
Both 32 and 64 bit transport the vector number through ORIG_[RE]AX in pt_regs. Further the vector number is forced to fit into an u8 and is complemented and offset by 0x80 so it's in the signed character range. Otherwise GAS would expand the pushq to a 5 byte instruction for any vector > 0x7F.
Treat the vector number like an error code and hand it to the C function as argument. This allows to get rid of the extra entry code in a later step.
Simplify the error code push magic by implementing the pushq imm8 via a '.byte 0x6a, vector' sequence so GAS is not able to screw it up. As the pushq imm8 is sign extending the resulting error code needs to be truncated to 8 bits in C code.
--- arch/x86/entry/calling.h | 5 ++- arch/x86/entry/entry_32.S | 33 ++------------------- arch/x86/entry/entry_64.S | 40 ++++--------------------- arch/x86/include/asm/entry_arch.h | 2 +- arch/x86/include/asm/hw_irq.h | 1 +- arch/x86/include/asm/idtentry.h | 48 ++++++++++++++++++++++++++++++- arch/x86/include/asm/irq.h | 2 +- arch/x86/include/asm/traps.h | 3 +- arch/x86/kernel/apic/apic.c | 31 +++++++++++++++---- arch/x86/kernel/idt.c | 2 +- arch/x86/kernel/irq.c | 14 +++++---- 11 files changed, 103 insertions(+), 78 deletions(-)
arch/x86/entry/calling.h +4 -1
@@ -341,7 +341,10 @@ For 32-bit we have the following conventions - kernel is built with#endif .endm -#endif /* CONFIG_X86_64 */ +#else /* CONFIG_X86_64 */ +# undef UNWIND_HINT_IRET_REGS +# define UNWIND_HINT_IRET_REGS +#endif /* !CONFIG_X86_64 */ .macro STACKLEAK_ERASE#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
arch/x86/entry/entry_32.S +4 -29
@@ -1215,40 +1215,15 @@ SYM_FUNC_END(entry_INT80_32) #endif .endm -/* - * Build the entry stubs with some assembler magic. - * We pack 1 stub into every 8-byte block. - */ - .align 8 -SYM_CODE_START(irq_entries_start) - vector=FIRST_EXTERNAL_VECTOR - .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) - pushl $(~vector+0x80) /* Note: always in signed byte range */ - vector=vector+1 - jmp common_interrupt - .align 8 - .endr -SYM_CODE_END(irq_entries_start) - #ifdef CONFIG_X86_LOCAL_APIC - .align 8 -SYM_CODE_START(spurious_entries_start) - vector=FIRST_SYSTEM_VECTOR - .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) - pushl $(~vector+0x80) /* Note: always in signed byte range */ - vector=vector+1 - jmp common_spurious - .align 8 - .endr -SYM_CODE_END(spurious_entries_start) - SYM_CODE_START_LOCAL(common_spurious) ASM_CLAC - addl $-0x80, (%esp) /* Adjust vector into the [-256, -1] range */ SAVE_ALL switch_stacks=1 ENCODE_FRAME_POINTER TRACE_IRQS_OFF movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ call smp_spurious_interrupt jmp ret_from_intr SYM_CODE_END(common_spurious)
@@ -1261,12 +1236,12 @@ SYM_CODE_END(common_spurious) .p2align CONFIG_X86_L1_CACHE_SHIFT SYM_CODE_START_LOCAL(common_interrupt) ASM_CLAC - addl $-0x80, (%esp) /* Adjust vector into the [-256, -1] range */ - SAVE_ALL switch_stacks=1 ENCODE_FRAME_POINTER TRACE_IRQS_OFF movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ call do_IRQ jmp ret_from_intr SYM_CODE_END(common_interrupt)
arch/x86/entry/entry_64.S +7 -33
@@ -358,34 +358,6 @@ SYM_CODE_START(ret_from_fork) SYM_CODE_END(ret_from_fork) .popsection -/* - * Build the entry stubs with some assembler magic. - * We pack 1 stub into every 8-byte block. - */ - .align 8 -SYM_CODE_START(irq_entries_start) - vector=FIRST_EXTERNAL_VECTOR - .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) - UNWIND_HINT_IRET_REGS - pushq $(~vector+0x80) /* Note: always in signed byte range */ - jmp common_interrupt - .align 8 - vector=vector+1 - .endr -SYM_CODE_END(irq_entries_start) - - .align 8 -SYM_CODE_START(spurious_entries_start) - vector=FIRST_SYSTEM_VECTOR - .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) - UNWIND_HINT_IRET_REGS - pushq $(~vector+0x80) /* Note: always in signed byte range */ - jmp common_spurious - .align 8 - vector=vector+1 - .endr -SYM_CODE_END(spurious_entries_start) - .macro DEBUG_ENTRY_ASSERT_IRQS_OFF #ifdef CONFIG_DEBUG_ENTRY pushq %rax
@@ -755,13 +727,14 @@ _ASM_NOKPROBE(interrupt_entry) /* Interrupt entry/exit. */ /* - * The interrupt stubs push (~vector+0x80) onto the stack and + * The interrupt stubs push vector onto the stack and * then jump to common_spurious/interrupt. */ SYM_CODE_START_LOCAL(common_spurious) - addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1 + movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ + movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ call smp_spurious_interrupt /* rdi points to pt_regs */ jmp ret_from_intr SYM_CODE_END(common_spurious)
@@ -770,10 +743,11 @@ _ASM_NOKPROBE(common_spurious) /* common_interrupt is a hotpath. Align it */ .p2align CONFIG_X86_L1_CACHE_SHIFT SYM_CODE_START_LOCAL(common_interrupt) - addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1 - call do_IRQ /* rdi points to pt_regs */ + movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ + movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ + call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY)
@@ -1022,7 +996,7 @@ apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif apicinterrupt ERROR_APIC_VECTOR error_interrupt smp_error_interrupt -apicinterrupt SPURIOUS_APIC_VECTOR spurious_interrupt smp_spurious_interrupt +apicinterrupt SPURIOUS_APIC_VECTOR spurious_apic_interrupt smp_spurious_apic_interrupt #ifdef CONFIG_IRQ_WORK apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt
arch/x86/include/asm/entry_arch.h +1 -1
@@ -35,7 +35,7 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR) BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR) BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR) -BUILD_INTERRUPT(spurious_interrupt,SPURIOUS_APIC_VECTOR) +BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR) BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR)#ifdef CONFIG_IRQ_WORK
arch/x86/include/asm/hw_irq.h +1 -0
@@ -39,6 +39,7 @@ extern asmlinkage void irq_work_interrupt(void); extern asmlinkage void uv_bau_message_intr1(void); extern asmlinkage void spurious_interrupt(void); +extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +48 -0
54 lines not on this page · raw message
arch/x86/include/asm/irq.h +1 -1
@@ -36,7 +36,7 @@ extern void native_init_IRQ(void);extern void handle_irq (struct irq_desc *desc,struct pt_regs *regs); -extern __visible void do_IRQ(struct pt_regs *regs); +extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);extern void init_ISA_irqs (void );
arch/x86/include/asm/traps.h +2 -1
@@ -41,8 +41,9 @@ asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);#endif void smp_apic_timer_interrupt (struct pt_regs *regs); -void smp_spurious_interrupt (struct pt_regs *regs);void smp_error_interrupt (struct pt_regs *regs); +void smp_spurious_apic_interrupt (struct pt_regs *regs); +void smp_spurious_interrupt (struct pt_regs *regs,unsigned long vector); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +25 -6
@@ -2153,15 +2153,29 @@ void __init register_lapic_address(unsigned long address)* Local APIC interrupts */ -/* -* This interrupt should _never_ happen with our APIC/SMP architecture +/** +* smp_spurious_interrupt - Catch all for interrupts raised on unused vectors +* @regs: Pointer to pt_regs on stack +* @error_code: The vector number is in the lower 8 bits +* +* This is invoked from ASM entry code to catch all interrupts which +* trigger on an entry which is routed to the common_spurious idtentry +* point. +* +* Also called from smp_spurious_apic_interrupt(). */ -__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs) +__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs, +unsigned long vector) { -u8 vector = ~regs->orig_ax;u32 v;entering_irq (); +/* +* The push in the entry ASM code which stores the vector number on +* the stack in the error code slot is sign expanding. Just use the +* lower 8 bits. +*/ + vector &=0xFF ;trace_spurious_apic_entry (vector);inc_irq_stat (irq_spurious_count);
@@ -2182,11 +2196,11 @@ __visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs)*/ v =apic_read (APIC_ISR + ((vector & ~0x1f ) >>1 ));if (v & (1 << (vector &0x1f ))) { -pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n" , +pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n" , vector,smp_processor_id ());ack_APIC_irq (); }else { -pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n" , +pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n" , vector,smp_processor_id ()); } out:
@@ -2194,6 +2208,11 @@ out:exiting_irq (); } +__visible void smp_spurious_apic_interrupt(struct pt_regs *regs) +{ +smp_spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +} +/* * This interrupt should never happen with our APIC/SMP architecture */
arch/x86/kernel/idt.c +1 -1
@@ -145,7 +145,7 @@ static const __initconst struct idt_data apic_idts[] = { #ifdef CONFIG_X86_UV INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), #endif - INTG(SPURIOUS_APIC_VECTOR, spurious_interrupt), + INTG(SPURIOUS_APIC_VECTOR, spurious_apic_interrupt), INTG(ERROR_APIC_VECTOR, error_interrupt), #endif };
arch/x86/kernel/irq.c +9 -5
@@ -227,14 +227,18 @@ u64 arch_irq_stat(void)* SMP cross-CPU interrupts have their own specific * handlers). */ -__visible void __irq_entry do_IRQ(struct pt_regs *regs) +__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) {struct pt_regs *old_regs =set_irq_regs (regs); -struct irq_desc * desc; -/* high bit used in ret_from_ code */ -unsigned vector = ~regs->orig_ax; +struct irq_desc *desc;entering_irq (); +/* +* The push in the entry ASM code which stores the vector number on +* the stack in the error code slot is sign expanding. Just use the +* lower 8 bits. +*/ + vector &=0xFF ;/* entering_irq() tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" );
@@ -249,7 +253,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs)ack_APIC_irq ();if (desc ==VECTOR_UNUSED ) { -pr_emerg_ratelimited ("%s: %d.%d No irq handler for vector\n" , +pr_emerg_ratelimited ("%s: %d.%lu No irq handler for vector\n" , __func__,smp_processor_id (), vector); }else {
Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
In reply to Thomas GleixnerHi Thomas,
Quoted 14 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Device interrupts which go through
do_IRQ()or the spurious interrupt handler have their separate entry code on 64 bit for no good reason.Both 32 and 64 bit transport the vector number through ORIG_[RE]AX in
pt_regs. Further the vector number is forced to fit into an u8 and is complemented and offset by0x80so it's in the signed character range. Otherwise GAS would expand the pushq to a 5 byte instruction for any vector >0x7F.Treat the vector number like an error code and hand it to the C function as argument. This allows to get rid of the extra entry code in a later step.
Quoted 8 lines
Simplify the error code push magic by implementing the pushq imm8 via a '.byte
0x6a, vector' sequence so GAS is not able to screw it up. As the pushq imm8 is sign extending the resulting error code needs to be truncated to 8 bits in C code.Originally-by Andy Lutomirski <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>Acked-by Andy Lutomirski <[email protected]>
I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:
https://github.com/scylladb/scylla/issues/7036
What we're seeing is that syscalls such as membarrier() take forever (0-10 µs would be normal):
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
53.26 12.458881 185953 67 membarrier
15.79 3.693651 17843 207 49 read 11.17 2.613350 67008 39 io_pgetevents 10.89 2.547772 11795 216 timerfd_settime 6.91 1.616802 11073 146 rt_sigprocmask 1.39 0.325955 3542 92 timer_settime 0.36 0.083691 526 159 io_submit 0.22 0.051399 535 96 write 0.00 0.000783 37 21 sendmsg 0.00 0.000057 3 18 9 ioctl
------ ----------- ----------- --------- --------- ----------------
100.00 23.392341 1061 58 total
That again seems to stem from a vastly slowed down smp_call_function_many_cond():
Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol 94.51% [kernel] [k] smp_call_function_many_cond 0.76% [kernel] [k] __do_softirq 0.32% [kernel] [k] native_queued_spin_lock_slowpath
[...]
which is stuck in
│ csd_lock_wait():
│ smp_cond_load_acquire(&csd->flags, !(VAL &
0.00 │ mov 0x8(%rcx),%edx
0.00 │ and $0x1,%edx
│ ↓ je 2b9
│ rep_nop():
0.70 │2af: pause
│ csd_lock_wait():
92.82 │ mov 0x8(%rcx),%edx
6.48 │ and $0x1,%edx
0.00 │ ↑ jne 2af
0.00 │2b9: ↑ jmp 282Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.
Do you have any further pointers I could look at?
Thanks,
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
Alex,
Quoted 8 lines · Alexander Graf
I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:
https://github.com/scylladb/scylla/issues/7036What we're seeing is that syscalls such as
membarrier()take forever (0-10 µs would be normal):
...
Quoted 26 lines
That again seems to stem from a vastly slowed down
smp_call_function_many_cond():Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol94.51% [kernel] [k] smp_call_function_many_cond 0.76% [kernel] [k] __do_softirq 0.32% [kernel] [k] native_queued_spin_lock_slowpath[...]
which is stuck in
│ csd_lock_wait(): │ smp_cond_load_acquire(&csd->flags, !(VAL & 0.00 │ mov 0x8(%rcx),%edx 0.00 │ and $0x1,%edx │ ↓ je 2b9 │ rep_nop(): 0.70 │2af: pause │ csd_lock_wait(): 92.82 │ mov 0x8(%rcx),%edx 6.48 │ and $0x1,%edx 0.00 │ ↑ jne 2af 0.00 │2b9: ↑ jmp 282
Quoted 2 lines
Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.
I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through do_IRQ() at all.
Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.
Thanks,
tglx
Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
Hi Thomas,
Quoted 48 lines · Thomas Gleixner
Alex,
Quoted 8 lines · Alexander Graf
I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:
https://github.com/scylladb/scylla/issues/7036What we're seeing is that syscalls such as
membarrier()take forever (0-10 µs would be normal):...
Quoted 26 lines
That again seems to stem from a vastly slowed down
smp_call_function_many_cond():Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol94.51% [kernel] [k] smp_call_function_many_cond 0.76% [kernel] [k] __do_softirq 0.32% [kernel] [k] native_queued_spin_lock_slowpath[...]
which is stuck in
│ csd_lock_wait(): │ smp_cond_load_acquire(&csd->flags, !(VAL & 0.00 │ mov 0x8(%rcx),%edx 0.00 │ and $0x1,%edx │ ↓ je 2b9 │ rep_nop(): 0.70 │2af: pause │ csd_lock_wait(): 92.82 │ mov 0x8(%rcx),%edx 6.48 │ and $0x1,%edx 0.00 │ ↑ jne 2af 0.00 │2b9: ↑ jmp 282Quoted 2 lines
Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.
I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through
do_IRQ()at all.
Quoted 3 lines
Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.
I am as puzzled as you are, but the bisect is very clear: 79b9c183021e works fast and 633260fa1 (as well as mainline) shows the weird behavior above.
It gets even better. This small (demonstrative only, mangled) patch on top of 633260fa1 also resolves the performance issue:
arch/x86/kernel/irq.c +1 -0
@@ -239,6 +239,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector)* lower 8 bits. */ vector &=0xFF ; + regs->orig_ax = ~vector;/* entering_irq() tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" );
To me that sounds like some irq exit code somewhere must still be looking at orig_ax to decide on something - and that something is wrong now that we removed the negation of the vector. It also seems to have an impact on remote function calls.
I'll have a deeper look tomorrow again if I can find any such place, but I wouldn't mind if anyone could point me into the right direction earlier :).
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Andy Lutomirski
maintainerRe: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
Quoted 5 lines · Alexander Graf
Hi Thomas,
Quoted 1 line · Thomas Gleixner
void
irq_complete_move(struct irq_cfg *cfg)
{
__irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}Quoted 73 lines
Quoted 47 lines
Alex,
Quoted 8 lines · Alexander Graf
I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:
https://github.com/scylladb/scylla/issues/7036What we're seeing is that syscalls such as
membarrier()take forever (0-10 µs would be normal):...
Quoted 26 lines
That again seems to stem from a vastly slowed down
smp_call_function_many_cond():Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol94.51% [kernel] [k] smp_call_function_many_cond 0.76% [kernel] [k] __do_softirq 0.32% [kernel] [k] native_queued_spin_lock_slowpath[...]
which is stuck in
│ csd_lock_wait(): │ smp_cond_load_acquire(&csd->flags, !(VAL & 0.00 │ mov 0x8(%rcx),%edx 0.00 │ and $0x1,%edx │ ↓ je 2b9 │ rep_nop(): 0.70 │2af: pause │ csd_lock_wait(): 92.82 │ mov 0x8(%rcx),%edx 6.48 │ and $0x1,%edx 0.00 │ ↑ jne 2af 0.00 │2b9: ↑ jmp 282Quoted 2 lines
Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.
I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through
do_IRQ()at all.Quoted 3 lines
Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.
I am as puzzled as you are, but the bisect is very clear:
79b9c183021eworks fast and633260fa1(as well as mainline) shows the weird behavior above.It gets even better. This small (demonstrative only, mangled) patch on top of
633260fa1also resolves the performance issue:arch/x86/kernel/irq.c +1
-0@@ -239,6 +239,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs*regs, unsigned long vector) * lower 8 bits. */ vector &=0xFF ; + regs->orig_ax = ~vector;/* entering_irq() tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
Quoted 8 lines
To me that sounds like some irq exit code somewhere must still be looking at
orig_axto decide on something - and that something is wrong now that we removed the negation of the vector. It also seems to have an impact on remote function calls.I'll have a deeper look tomorrow again if I can find any such place, but I wouldn't mind if anyone could point me into the right direction earlier :).
How about this:
void irq_complete_move(struct irq_cfg *cfg)
{
__irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}in arch/x86/kernel/apic/vector.c.
Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
Hi Andy,
Quoted 95 lines · Andy Lutomirski
Quoted 5 lines · Alexander Graf
Hi Thomas,
Quoted 1 line · Thomas Gleixner
void
irq_complete_move(struct irq_cfg *cfg){ __irq_complete_move(cfg, ~get_irq_regs()->orig_ax); }Quoted 73 lines
Quoted 47 lines
Alex,
Quoted 8 lines · Alexander Graf
I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:
https://github.com/scylladb/scylla/issues/7036What we're seeing is that syscalls such as
membarrier()take forever (0-10 µs would be normal):...
Quoted 26 lines
That again seems to stem from a vastly slowed down
smp_call_function_many_cond():Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol94.51% [kernel] [k] smp_call_function_many_cond 0.76% [kernel] [k] __do_softirq 0.32% [kernel] [k] native_queued_spin_lock_slowpath[...]
which is stuck in
│ csd_lock_wait(): │ smp_cond_load_acquire(&csd->flags, !(VAL & 0.00 │ mov 0x8(%rcx),%edx 0.00 │ and $0x1,%edx │ ↓ je 2b9 │ rep_nop(): 0.70 │2af: pause │ csd_lock_wait(): 92.82 │ mov 0x8(%rcx),%edx 6.48 │ and $0x1,%edx 0.00 │ ↑ jne 2af 0.00 │2b9: ↑ jmp 282Quoted 2 lines
Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.
I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through
do_IRQ()at all.Quoted 3 lines
Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.
I am as puzzled as you are, but the bisect is very clear:
79b9c183021eworks fast and633260fa1(as well as mainline) shows the weird behavior above.It gets even better. This small (demonstrative only, mangled) patch on top of
633260fa1also resolves the performance issue:arch/x86/kernel/irq.c +1
-0@@ -239,6 +239,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs*regs, unsigned long vector) * lower 8 bits. */ vector &=0xFF ; + regs->orig_ax = ~vector;/* entering_irq() tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");Quoted 8 lines
To me that sounds like some irq exit code somewhere must still be looking at
orig_axto decide on something - and that something is wrong now that we removed the negation of the vector. It also seems to have an impact on remote function calls.I'll have a deeper look tomorrow again if I can find any such place, but I wouldn't mind if anyone could point me into the right direction earlier :).
How about this:
Quoted 7 lines
void
irq_complete_move(struct irq_cfg *cfg){ __irq_complete_move(cfg, ~get_irq_regs()->orig_ax); }in
arch/x86/kernel/apic/vector.c.
Thanks a lot, I stumbled over the same thing just after I sent the email as well and had been trying to see if I can quickly patch it up before I fall asleep :).
The code path above is used by the APIC vector move (irqbalance) logic, which explains why not everyone was seeing issues.
So with 633260fa1 applied, we never get out of moving state for our IRQ because orig_ax is always -1. That means we send an IPI to the cleanup vector on every single device interrupt, completely occupying the poor CPU that we moved the IRQ from.
I've confirmed that the patch below fixes the issue and will send a proper, more complete patch on top of mainline with fancy description and stable tag tomorrow.
Alex
arch/x86/entry/entry_64.S +0 -2
@@ -734,7 +734,6 @@ SYM_CODE_START_LOCAL(common_spurious) call interrupt_entry UNWIND_HINT_REGS indirect=1 movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ call smp_spurious_interrupt /* rdi points to pt_regs */ jmp ret_from_intr SYM_CODE_END(common_spurious)
@@ -746,7 +745,6 @@ SYM_CODE_START_LOCAL(common_interrupt) call interrupt_entry UNWIND_HINT_REGS indirect=1 movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr:
arch/x86/kernel/apic/vector.c +1 -1
@@ -934,7 +934,7 @@ static void __irq_complete_move(struct irq_cfg *cfg, unsigned vector)void irq_complete_move (struct irq_cfg *cfg) { -__irq_complete_move (cfg, ~get_irq_regs ()->orig_ax); +__irq_complete_move (cfg,get_irq_regs ()->orig_ax); }/*
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
Quoted 153 lines · Alexander Graf
Hi Andy,
Quoted 95 lines · Andy Lutomirski
Quoted 5 lines · Alexander Graf
Hi Thomas,
Quoted 1 line · Thomas Gleixner
void
irq_complete_move(struct irq_cfg *cfg){ __irq_complete_move(cfg, ~get_irq_regs()->orig_ax); }Quoted 73 lines
Quoted 47 lines
Alex,
Quoted 8 lines · Alexander Graf
I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:
https://github.com/scylladb/scylla/issues/7036What we're seeing is that syscalls such as
membarrier()take forever (0-10 µs would be normal):...
Quoted 26 lines
That again seems to stem from a vastly slowed down
smp_call_function_many_cond():Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol94.51% [kernel] [k] smp_call_function_many_cond 0.76% [kernel] [k] __do_softirq 0.32% [kernel] [k] native_queued_spin_lock_slowpath[...]
which is stuck in
│ csd_lock_wait(): │ smp_cond_load_acquire(&csd->flags, !(VAL & 0.00 │ mov 0x8(%rcx),%edx 0.00 │ and $0x1,%edx │ ↓ je 2b9 │ rep_nop(): 0.70 │2af: pause │ csd_lock_wait(): 92.82 │ mov 0x8(%rcx),%edx 6.48 │ and $0x1,%edx 0.00 │ ↑ jne 2af 0.00 │2b9: ↑ jmp 282Quoted 2 lines
Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.
I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through
do_IRQ()at all.Quoted 3 lines
Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.
I am as puzzled as you are, but the bisect is very clear:
79b9c183021eworks fast and633260fa1(as well as mainline) shows the weird behavior above.It gets even better. This small (demonstrative only, mangled) patch on top of
633260fa1also resolves the performance issue:arch/x86/kernel/irq.c +1
-0@@ -239,6 +239,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs*regs, unsigned long vector) * lower 8 bits. */ vector &=0xFF ; + regs->orig_ax = ~vector;/* entering_irq() tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");Quoted 8 lines
To me that sounds like some irq exit code somewhere must still be looking at
orig_axto decide on something - and that something is wrong now that we removed the negation of the vector. It also seems to have an impact on remote function calls.I'll have a deeper look tomorrow again if I can find any such place, but I wouldn't mind if anyone could point me into the right direction earlier :).
How about this:
Quoted 7 lines
void
irq_complete_move(struct irq_cfg *cfg){ __irq_complete_move(cfg, ~get_irq_regs()->orig_ax); }in
arch/x86/kernel/apic/vector.c.Thanks a lot, I stumbled over the same thing just after I sent the email as well and had been trying to see if I can quickly patch it up before I fall asleep :).
The code path above is used by the APIC vector move (irqbalance) logic, which explains why not everyone was seeing issues.
So with
633260fa1applied, we never get out of moving state for our IRQ becauseorig_axis always -1. That means we send an IPI to the cleanup vector on every single device interrupt, completely occupying the poor CPU that we moved the IRQ from.I've confirmed that the patch below fixes the issue and will send a proper, more complete patch on top of mainline with fancy description and stable tag tomorrow.
Alex
arch/x86/entry/entry_64.S +0
-2@@ -734,7 +734,6 @@ SYM_CODE_START_LOCAL(common_spurious) call interrupt_entry UNWIND_HINT_REGS indirect=1 movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ call smp_spurious_interrupt /* rdi points to pt_regs */ jmp ret_from_intr SYM_CODE_END(common_spurious)@@ -746,7 +745,6 @@ SYM_CODE_START_LOCAL(common_interrupt) call interrupt_entry UNWIND_HINT_REGS indirect=1 movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr:arch/x86/kernel/apic/vector.c +0
-0
arch/x86/kernel/apic/vector.c static void __irq_complete_move(struct irq_cfg *cfg, · line 939
Quoted 7 lines
arch/x86/kernel/apic/vector.c +1
-1@@ -934,7 +934,7 @@ static void __irq_complete_move(struct irq_cfg *cfg, unsigned vector)void irq_complete_move (struct irq_cfg *cfg) { -__irq_complete_move (cfg, ~get_irq_regs ()->orig_ax); +__irq_complete_move (cfg,get_irq_regs ()->orig_ax); }
I think you need to also truncate the vector to 8-bits, since it now gets sign-extended when pushed into the orig_ax slot.
--
Brian Gerst
Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs
In reply to Andy LutomirskiQuoted 8 lines · Andy Lutomirski
How about this:
void
irq_complete_move(struct irq_cfg *cfg){ __irq_complete_move(cfg, ~get_irq_regs()->orig_ax); }in
arch/x86/kernel/apic/vector.c.
Duh....
PATCH v9 22/39 x86/irq: Rework handle_irq() for 64bit
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
To consolidate the interrupt entry/exit code vs. the other exceptions make handle_irq() and inline and handle both 64bit and 32bit
Preparatory change to move irq stack switching for 64bit to C which allows to consolidate the entry exit handling by reusing the idtentry machinery both in ASM and C.
---
V9: Adopt to inlined run_cond_on_irqstack()
--- arch/x86/include/asm/irq.h | 2 +- arch/x86/kernel/irq.c | 11 ++++++++++- arch/x86/kernel/irq_32.c | 2 +- 3 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/include/asm/irq.h +1 -1
@@ -34,7 +34,7 @@ extern __visible void smp_kvm_posted_intextern void (*x86_platform_ipi_callback)(void );extern void native_init_IRQ (void ); -extern void handle_irq (struct irq_desc *desc,struct pt_regs *regs); +extern void __handle_irq (struct irq_desc *desc,struct pt_regs *regs); extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);
arch/x86/kernel/irq.c +10 -1
@@ -13,6 +13,7 @@#include <linux/export.h> #include <linux/irq.h> +#include <asm/irq_stack.h> #include <asm/apic.h> #include <asm/io_apic.h> #include <asm/irq.h>
@@ -221,6 +222,14 @@ u64 arch_irq_stat(void)return sum; } +static __always_inline void handle_irq(struct irq_desc *desc, +struct pt_regs *regs) +{ +if (IS_ENABLED (CONFIG_X86_64 )) +run_on_irqstack_cond (desc->handle_irq, desc, regs); +else +__handle_irq (desc, regs); +}/* * do_IRQ handles all normal device IRQ's (the special
@@ -246,7 +255,7 @@ u64 arch_irq_stat(void) desc =__this_cpu_read (vector_irq[vector]);if (likely (!IS_ERR_OR_NULL (desc))) {if (IS_ENABLED (CONFIG_X86_32 )) -handle_irq (desc, regs); +__handle_irq (desc, regs);else generic_handle_irq_desc (desc); }else {
arch/x86/kernel/irq_32.c +1 -1
@@ -148,7 +148,7 @@ void do_softirq_own_stack(void)call_on_stack (__do_softirq, isp); } -void handle_irq (struct irq_desc *desc,struct pt_regs *regs) +void __handle_irq (struct irq_desc *desc,struct pt_regs *regs) {int overflow =check_stack_overflow ();
[tip: x86/entry] x86/irq: Rework handle_irq() for 64-bit
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 9fe3411558d31f9489cd58400b2f508a98dc13a5 Gitweb: https://git.kernel.org/tip/9fe3411558d31f9489cd58400b2f508a98dc13a5 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:35 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/irq: Rework handle_irq() for 64-bit
To consolidate the interrupt entry/exit code vs. the other exceptions make handle_irq() and inline and handle both 64-bit and 32-bit mode.
Preparatory change to move irq stack switching for 64-bit to C which allows to consolidate the entry exit handling by reusing the idtentry machinery both in ASM and C.
--- arch/x86/include/asm/irq.h | 2 +- arch/x86/kernel/irq.c | 11 ++++++++++- arch/x86/kernel/irq_32.c | 2 +- 3 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/include/asm/irq.h +1 -1
@@ -34,7 +34,7 @@ extern __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs);extern void (*x86_platform_ipi_callback)(void );extern void native_init_IRQ (void ); -extern void handle_irq (struct irq_desc *desc,struct pt_regs *regs); +extern void __handle_irq (struct irq_desc *desc,struct pt_regs *regs); extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);
arch/x86/kernel/irq.c +10 -1
@@ -13,6 +13,7 @@#include <linux/export.h> #include <linux/irq.h> +#include <asm/irq_stack.h> #include <asm/apic.h> #include <asm/io_apic.h> #include <asm/irq.h>
@@ -221,6 +222,14 @@ u64 arch_irq_stat(void)return sum; } +static __always_inline void handle_irq(struct irq_desc *desc, +struct pt_regs *regs) +{ +if (IS_ENABLED (CONFIG_X86_64 )) +run_on_irqstack_cond (desc->handle_irq, desc, regs); +else +__handle_irq (desc, regs); +}/* * do_IRQ handles all normal device IRQ's (the special
@@ -246,7 +255,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) desc =__this_cpu_read (vector_irq[vector]);if (likely (!IS_ERR_OR_NULL (desc))) {if (IS_ENABLED (CONFIG_X86_32 )) -handle_irq (desc, regs); +__handle_irq (desc, regs);else generic_handle_irq_desc (desc); }else {
arch/x86/kernel/irq_32.c +1 -1
@@ -148,7 +148,7 @@ void do_softirq_own_stack(void)call_on_stack (__do_softirq, isp); } -void handle_irq (struct irq_desc *desc,struct pt_regs *regs) +void __handle_irq (struct irq_desc *desc,struct pt_regs *regs) {int overflow =check_stack_overflow ();
PATCH v9 23/39 x86/entry: Add IRQENTRY_IRQ macro
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Provide a seperate IDTENTRY macro for device interrupts. Similar to IDTENTRY_ERRORCODE with the addition of invoking irq_enter/exit_rcu() and providing the errorcode as a 'u8' argument to the C function, which truncates the sign extended vector number.
---
V9: Use cond RCU handling
--- arch/x86/entry/entry_32.S | 14 +++++++++++ arch/x86/entry/entry_64.S | 14 +++++++++++ arch/x86/include/asm/idtentry.h | 48 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+)
arch/x86/entry/entry_32.S +14 -0
@@ -751,6 +751,20 @@ SYM_CODE_START(\asmsym) SYM_CODE_END(\asmsym) .endm +.macro idtentry_irq vector cfunc + .p2align CONFIG_X86_L1_CACHE_SHIFT +SYM_CODE_START_LOCAL(asm_\cfunc) + ASM_CLAC + SAVE_ALL switch_stacks=1 + ENCODE_FRAME_POINTER + movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ + call \cfunc + jmp handle_exception_return +SYM_CODE_END(asm_\cfunc) +.endm + /* * Include the defines which emit the idt entries which are shared * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +14 -0
@@ -528,6 +528,20 @@ SYM_CODE_END(\asmsym) .endm /* + * Interrupt entry/exit. + * + + The interrupt stubs push (vector) onto the stack, which is the error_code + * position of idtentry exceptions, and jump to one of the two idtentry points + * (common/spurious). + * + * common_interrupt is a hotpath, align it to a cache line + */ +.macro idtentry_irq vector cfunc + .p2align CONFIG_X86_L1_CACHE_SHIFT + idtentry \vector asm_\cfunc \cfunc has_error_code=1 +.endm + +/* * MCE and DB exceptions */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +48 -0
60 lines not on this page · raw message
[tip: x86/entry] x86/entry: Add IRQENTRY_IRQ macro
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: f7376b1a72bb4bcfe2212150e3cf79f21cf88912 Gitweb: https://git.kernel.org/tip/f7376b1a72bb4bcfe2212150e3cf79f21cf88912 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:36 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Add IRQENTRY_IRQ macro
Provide a seperate IDTENTRY macro for device interrupts. Similar to IDTENTRY_ERRORCODE with the addition of invoking irq_enter/exit_rcu() and providing the errorcode as a 'u8' argument to the C function, which truncates the sign extended vector number.
--- arch/x86/entry/entry_32.S | 14 +++++++++- arch/x86/entry/entry_64.S | 14 +++++++++- arch/x86/include/asm/idtentry.h | 48 ++++++++++++++++++++++++++++++++- 3 files changed, 76 insertions(+)
arch/x86/entry/entry_32.S +14 -0
@@ -751,6 +751,20 @@ SYM_CODE_START(\asmsym) SYM_CODE_END(\asmsym) .endm +.macro idtentry_irq vector cfunc + .p2align CONFIG_X86_L1_CACHE_SHIFT +SYM_CODE_START_LOCAL(asm_\cfunc) + ASM_CLAC + SAVE_ALL switch_stacks=1 + ENCODE_FRAME_POINTER + movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ + call \cfunc + jmp handle_exception_return +SYM_CODE_END(asm_\cfunc) +.endm + /* * Include the defines which emit the idt entries which are shared * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +14 -0
@@ -528,6 +528,20 @@ SYM_CODE_END(\asmsym) .endm /* + * Interrupt entry/exit. + * + + The interrupt stubs push (vector) onto the stack, which is the error_code + * position of idtentry exceptions, and jump to one of the two idtentry points + * (common/spurious). + * + * common_interrupt is a hotpath, align it to a cache line + */ +.macro idtentry_irq vector cfunc + .p2align CONFIG_X86_L1_CACHE_SHIFT + idtentry \vector asm_\cfunc \cfunc has_error_code=1 +.endm + +/* * MCE and DB exceptions */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +48 -0
60 lines not on this page · raw message
PATCH v9 24/39 x86/entry: Use idtentry for interrupts
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Replace the extra interrupt handling code and reuse the existing idtentry machinery. This moves the irq stack switching on 64 bit from ASM to C code; 32bit already does the stack switching in C.
--- arch/x86/entry/entry_32.S | 31 ------------------------------- arch/x86/entry/entry_64.S | 31 +++---------------------------- arch/x86/include/asm/hw_irq.h | 1 - arch/x86/include/asm/idtentry.h | 10 ++++++++-- arch/x86/include/asm/traps.h | 1 - arch/x86/kernel/apic/apic.c | 23 ++++++++--------------- arch/x86/kernel/apic/msi.c | 3 ++- arch/x86/kernel/irq.c | 27 +++++++-------------------- 8 files changed, 28 insertions(+), 99 deletions(-)
arch/x86/entry/entry_32.S +0 -31
@@ -1229,37 +1229,6 @@ SYM_FUNC_END(entry_INT80_32) #endif .endm -#ifdef CONFIG_X86_LOCAL_APIC -SYM_CODE_START_LOCAL(common_spurious) - ASM_CLAC - SAVE_ALL switch_stacks=1 - ENCODE_FRAME_POINTER - TRACE_IRQS_OFF - movl %esp, %eax - movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ - movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ - call smp_spurious_interrupt - jmp ret_from_intr -SYM_CODE_END(common_spurious) -#endif - -/* - * the CPU automatically disables interrupts when executing an IRQ vector, - * so IRQ-flags tracing has to follow that: - */ - .p2align CONFIG_X86_L1_CACHE_SHIFT -SYM_CODE_START_LOCAL(common_interrupt) - ASM_CLAC - SAVE_ALL switch_stacks=1 - ENCODE_FRAME_POINTER - TRACE_IRQS_OFF - movl %esp, %eax - movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ - movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ - call do_IRQ - jmp ret_from_intr -SYM_CODE_END(common_interrupt) - #define BUILD_INTERRUPT3(name, nr, fn) \ SYM_FUNC_START(name) \ ASM_CLAC; \
arch/x86/entry/entry_64.S +3 -28
@@ -737,32 +737,7 @@ SYM_CODE_START(interrupt_entry) SYM_CODE_END(interrupt_entry) _ASM_NOKPROBE(interrupt_entry) - -/* Interrupt entry/exit. */ - -/* - * The interrupt stubs push vector onto the stack and - * then jump to common_spurious/interrupt. - */ -SYM_CODE_START_LOCAL(common_spurious) - call interrupt_entry - UNWIND_HINT_REGS indirect=1 - movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ - call smp_spurious_interrupt /* rdi points to pt_regs */ - jmp ret_from_intr -SYM_CODE_END(common_spurious) -_ASM_NOKPROBE(common_spurious) - -/* common_interrupt is a hotpath. Align it */ - .p2align CONFIG_X86_L1_CACHE_SHIFT -SYM_CODE_START_LOCAL(common_interrupt) - call interrupt_entry - UNWIND_HINT_REGS indirect=1 - movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ - call do_IRQ /* rdi points to pt_regs */ - /* 0(%rsp): old RSP */ +SYM_CODE_START_LOCAL(common_interrupt_return) ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF
@@ -945,8 +920,8 @@ SYM_INNER_LABEL(native_irq_return_iret, */ jmp native_irq_return_iret #endif -SYM_CODE_END(common_interrupt) -_ASM_NOKPROBE(common_interrupt) +SYM_CODE_END(common_interrupt_return) +_ASM_NOKPROBE(common_interrupt_return) /* * APIC interrupts.
arch/x86/include/asm/hw_irq.h +0 -1
@@ -38,7 +38,6 @@ extern asmlinkage void error_interrupt(v extern asmlinkage void irq_work_interrupt(void); extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void spurious_interrupt(void); extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +8 -2
@@ -395,7 +395,7 @@ SYM_CODE_START(irq_entries_start) .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) UNWIND_HINT_IRET_REGS .byte 0x6a, vector - jmp common_interrupt + jmp asm_common_interrupt .nops (pos + 8) - . pos=pos+8 vector=vector+1
@@ -410,7 +410,7 @@ SYM_CODE_START(spurious_entries_start) .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) UNWIND_HINT_IRET_REGS .byte 0x6a, vector - jmp common_spurious + jmp asm_spurious_interrupt .nops (pos + 8) - . pos=pos+8 vector=vector+1
@@ -480,6 +480,12 @@ DECLARE_IDTENTRY_DF(X86_TRAP_DF, exc_douDECLARE_IDTENTRY (X86_TRAP_OTHER , exc_xen_hypervisor_callback);#endif +/* Device interrupts common/spurious */ +DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , common_interrupt); +#ifdef CONFIG_X86_LOCAL_APIC +DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , spurious_interrupt); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/traps.h +0 -1
@@ -43,7 +43,6 @@ asmlinkage void smp_deferred_error_intervoid smp_apic_timer_interrupt (struct pt_regs *regs);void smp_error_interrupt (struct pt_regs *regs);void smp_spurious_apic_interrupt (struct pt_regs *regs); -void smp_spurious_interrupt (struct pt_regs *regs,unsigned long vector); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +8 -15
@@ -2154,9 +2154,9 @@ void __init register_lapic_address(unsig*/ /** -* smp_spurious_interrupt - Catch all for interrupts raised on unused vectors +* spurious_interrupt - Catch all for interrupts raised on unused vectors * @regs: Pointer to pt_regs on stack -* @error_code: The vector number is in the lower 8 bits +* @vector: The vector number * * This is invoked from ASM entry code to catch all interrupts which * trigger on an entry which is routed to the common_spurious idtentry
@@ -2164,18 +2164,10 @@ void __init register_lapic_address(unsig* * Also called from smp_spurious_apic_interrupt(). */ -__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs, -unsigned long vector) +DEFINE_IDTENTRY_IRQ (spurious_interrupt) {u32 v; -entering_irq (); -/* -* The push in the entry ASM code which stores the vector number on -* the stack in the error code slot is sign expanding. Just use the -* lower 8 bits. -*/ - vector &=0xFF ;trace_spurious_apic_entry (vector);inc_irq_stat (irq_spurious_count);
@@ -2196,21 +2188,22 @@ void __init register_lapic_address(unsig*/ v =apic_read (APIC_ISR + ((vector & ~0x1f ) >>1 ));if (v & (1 << (vector &0x1f ))) { -pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n" , +pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n" , vector,smp_processor_id ());ack_APIC_irq (); }else { -pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n" , +pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n" , vector,smp_processor_id ()); } out:trace_spurious_apic_exit (vector); -exiting_irq (); } __visible void smp_spurious_apic_interrupt(struct pt_regs *regs) { -smp_spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +entering_irq (); +__spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +exiting_irq (); }/*
arch/x86/kernel/apic/msi.c +2 -1
@@ -115,7 +115,8 @@ msi_set_affinity(struct irq_data *irqd,* denote it as spurious which is no harm as this is a rare event * and interrupt handlers have to cope with spurious interrupts * anyway. If the vector is unused, then it is marked so it won't -* trigger the 'No irq handler for vector' warning in do_IRQ(). +* trigger the 'No irq handler for vector' warning in +* common_interrupt(). * * This requires to hold vector lock to prevent concurrent updates to * the affected vector.
arch/x86/kernel/irq.c +7 -20
@@ -20,6 +20,7 @@#include <asm/mce.h> #include <asm/hw_irq.h> #include <asm/desc.h> +#include <asm/traps.h> #define CREATE_TRACE_POINTS #include <asm/trace/irq_vectors.h>
@@ -232,37 +233,25 @@ static __always_inline void handle_irq(s }/* -* do_IRQ handles all normal device IRQ's (the special -* SMP cross-CPU interrupts have their own specific -* handlers). +* common_interrupt() handles all normal device IRQ's (the special SMP +* cross-CPU interrupts have their own entry points). */ -__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) +DEFINE_IDTENTRY_IRQ (common_interrupt) {struct pt_regs *old_regs =set_irq_regs (regs);struct irq_desc *desc; -entering_irq (); -/* -* The push in the entry ASM code which stores the vector number on -* the stack in the error code slot is sign expanding. Just use the -* lower 8 bits. -*/ - vector &=0xFF ; - -/* entering_irq() tells RCU that we're not quiescent. Check it. */ +/* entry code tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" ); desc =__this_cpu_read (vector_irq[vector]);if (likely (!IS_ERR_OR_NULL (desc))) { -if (IS_ENABLED (CONFIG_X86_32 )) -__handle_irq (desc, regs); -else -generic_handle_irq_desc (desc); +handle_irq (desc, regs); }else {ack_APIC_irq ();if (desc ==VECTOR_UNUSED ) { -pr_emerg_ratelimited ("%s: %d.%lu No irq handler for vector\n" , +pr_emerg_ratelimited ("%s: %d.%u No irq handler for vector\n" , __func__,smp_processor_id (), vector); }else {
@@ -270,8 +259,6 @@ static __always_inline void handle_irq(s } } -exiting_irq (); -set_irq_regs (old_regs); }
[tip: x86/entry] x86/entry: Use idtentry for interrupts
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: e1177738020f5ad104b5a5c31c78b1b4252deaca Gitweb: https://git.kernel.org/tip/e1177738020f5ad104b5a5c31c78b1b4252deaca Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:37 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Use idtentry for interrupts
Replace the extra interrupt handling code and reuse the existing idtentry machinery. This moves the irq stack switching on 64-bit from ASM to C code; 32-bit already does the stack switching in C.
--- arch/x86/entry/entry_32.S | 31 ------------------------------- arch/x86/entry/entry_64.S | 31 +++---------------------------- arch/x86/include/asm/hw_irq.h | 1 - arch/x86/include/asm/idtentry.h | 10 ++++++++-- arch/x86/include/asm/traps.h | 1 - arch/x86/kernel/apic/apic.c | 23 ++++++++--------------- arch/x86/kernel/apic/msi.c | 3 ++- arch/x86/kernel/irq.c | 27 +++++++-------------------- 8 files changed, 28 insertions(+), 99 deletions(-)
arch/x86/entry/entry_32.S +0 -31
@@ -1229,37 +1229,6 @@ SYM_FUNC_END(entry_INT80_32) #endif .endm -#ifdef CONFIG_X86_LOCAL_APIC -SYM_CODE_START_LOCAL(common_spurious) - ASM_CLAC - SAVE_ALL switch_stacks=1 - ENCODE_FRAME_POINTER - TRACE_IRQS_OFF - movl %esp, %eax - movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ - movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ - call smp_spurious_interrupt - jmp ret_from_intr -SYM_CODE_END(common_spurious) -#endif - -/* - * the CPU automatically disables interrupts when executing an IRQ vector, - * so IRQ-flags tracing has to follow that: - */ - .p2align CONFIG_X86_L1_CACHE_SHIFT -SYM_CODE_START_LOCAL(common_interrupt) - ASM_CLAC - SAVE_ALL switch_stacks=1 - ENCODE_FRAME_POINTER - TRACE_IRQS_OFF - movl %esp, %eax - movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ - movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ - call do_IRQ - jmp ret_from_intr -SYM_CODE_END(common_interrupt) - #define BUILD_INTERRUPT3(name, nr, fn) \ SYM_FUNC_START(name) \ ASM_CLAC; \
arch/x86/entry/entry_64.S +3 -28
@@ -737,32 +737,7 @@ SYM_CODE_START(interrupt_entry) SYM_CODE_END(interrupt_entry) _ASM_NOKPROBE(interrupt_entry) - -/* Interrupt entry/exit. */ - -/* - * The interrupt stubs push vector onto the stack and - * then jump to common_spurious/interrupt. - */ -SYM_CODE_START_LOCAL(common_spurious) - call interrupt_entry - UNWIND_HINT_REGS indirect=1 - movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ - call smp_spurious_interrupt /* rdi points to pt_regs */ - jmp ret_from_intr -SYM_CODE_END(common_spurious) -_ASM_NOKPROBE(common_spurious) - -/* common_interrupt is a hotpath. Align it */ - .p2align CONFIG_X86_L1_CACHE_SHIFT -SYM_CODE_START_LOCAL(common_interrupt) - call interrupt_entry - UNWIND_HINT_REGS indirect=1 - movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ - call do_IRQ /* rdi points to pt_regs */ - /* 0(%rsp): old RSP */ +SYM_CODE_START_LOCAL(common_interrupt_return) ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF
@@ -945,8 +920,8 @@ native_irq_return_ldt: */ jmp native_irq_return_iret #endif -SYM_CODE_END(common_interrupt) -_ASM_NOKPROBE(common_interrupt) +SYM_CODE_END(common_interrupt_return) +_ASM_NOKPROBE(common_interrupt_return) /* * APIC interrupts.
arch/x86/include/asm/hw_irq.h +0 -1
@@ -38,7 +38,6 @@ extern asmlinkage void error_interrupt(void); extern asmlinkage void irq_work_interrupt(void); extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void spurious_interrupt(void); extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +8 -2
@@ -395,7 +395,7 @@ SYM_CODE_START(irq_entries_start) .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) UNWIND_HINT_IRET_REGS .byte 0x6a, vector - jmp common_interrupt + jmp asm_common_interrupt nop/* Ensure that the above is 8 bytes max */ . = pos + 8
@@ -412,7 +412,7 @@ SYM_CODE_START(spurious_entries_start) .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) UNWIND_HINT_IRET_REGS .byte 0x6a, vector - jmp common_spurious + jmp asm_spurious_interrupt nop/* Ensure that the above is 8 bytes max */ . = pos + 8
@@ -484,6 +484,12 @@ DECLARE_IDTENTRY_DF(X86_TRAP_DF, exc_double_fault);DECLARE_IDTENTRY (X86_TRAP_OTHER , exc_xen_hypervisor_callback);#endif +/* Device interrupts common/spurious */ +DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , common_interrupt); +#ifdef CONFIG_X86_LOCAL_APIC +DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , spurious_interrupt); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/traps.h +0 -1
@@ -43,7 +43,6 @@ asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);void smp_apic_timer_interrupt (struct pt_regs *regs);void smp_error_interrupt (struct pt_regs *regs);void smp_spurious_apic_interrupt (struct pt_regs *regs); -void smp_spurious_interrupt (struct pt_regs *regs,unsigned long vector); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +8 -15
@@ -2154,9 +2154,9 @@ void __init register_lapic_address(unsigned long address)*/ /** -* smp_spurious_interrupt - Catch all for interrupts raised on unused vectors +* spurious_interrupt - Catch all for interrupts raised on unused vectors * @regs: Pointer to pt_regs on stack -* @error_code: The vector number is in the lower 8 bits +* @vector: The vector number * * This is invoked from ASM entry code to catch all interrupts which * trigger on an entry which is routed to the common_spurious idtentry
@@ -2164,18 +2164,10 @@ void __init register_lapic_address(unsigned long address)* * Also called from smp_spurious_apic_interrupt(). */ -__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs, -unsigned long vector) +DEFINE_IDTENTRY_IRQ (spurious_interrupt) {u32 v; -entering_irq (); -/* -* The push in the entry ASM code which stores the vector number on -* the stack in the error code slot is sign expanding. Just use the -* lower 8 bits. -*/ - vector &=0xFF ;trace_spurious_apic_entry (vector);inc_irq_stat (irq_spurious_count);
@@ -2196,21 +2188,22 @@ __visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs,*/ v =apic_read (APIC_ISR + ((vector & ~0x1f ) >>1 ));if (v & (1 << (vector &0x1f ))) { -pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n" , +pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n" , vector,smp_processor_id ());ack_APIC_irq (); }else { -pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n" , +pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n" , vector,smp_processor_id ()); } out:trace_spurious_apic_exit (vector); -exiting_irq (); } __visible void smp_spurious_apic_interrupt(struct pt_regs *regs) { -smp_spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +entering_irq (); +__spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +exiting_irq (); }/*
arch/x86/kernel/apic/msi.c +2 -1
@@ -115,7 +115,8 @@ msi_set_affinity(struct irq_data *irqd, const struct cpumask *mask, bool force)* denote it as spurious which is no harm as this is a rare event * and interrupt handlers have to cope with spurious interrupts * anyway. If the vector is unused, then it is marked so it won't -* trigger the 'No irq handler for vector' warning in do_IRQ(). +* trigger the 'No irq handler for vector' warning in +* common_interrupt(). * * This requires to hold vector lock to prevent concurrent updates to * the affected vector.
arch/x86/kernel/irq.c +7 -20
@@ -20,6 +20,7 @@#include <asm/mce.h> #include <asm/hw_irq.h> #include <asm/desc.h> +#include <asm/traps.h> #define CREATE_TRACE_POINTS #include <asm/trace/irq_vectors.h>
@@ -232,37 +233,25 @@ static __always_inline void handle_irq(struct irq_desc *desc, }/* -* do_IRQ handles all normal device IRQ's (the special -* SMP cross-CPU interrupts have their own specific -* handlers). +* common_interrupt() handles all normal device IRQ's (the special SMP +* cross-CPU interrupts have their own entry points). */ -__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) +DEFINE_IDTENTRY_IRQ (common_interrupt) {struct pt_regs *old_regs =set_irq_regs (regs);struct irq_desc *desc; -entering_irq (); -/* -* The push in the entry ASM code which stores the vector number on -* the stack in the error code slot is sign expanding. Just use the -* lower 8 bits. -*/ - vector &=0xFF ; - -/* entering_irq() tells RCU that we're not quiescent. Check it. */ +/* entry code tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" ); desc =__this_cpu_read (vector_irq[vector]);if (likely (!IS_ERR_OR_NULL (desc))) { -if (IS_ENABLED (CONFIG_X86_32 )) -__handle_irq (desc, regs); -else -generic_handle_irq_desc (desc); +handle_irq (desc, regs); }else {ack_APIC_irq ();if (desc ==VECTOR_UNUSED ) { -pr_emerg_ratelimited ("%s: %d.%lu No irq handler for vector\n" , +pr_emerg_ratelimited ("%s: %d.%u No irq handler for vector\n" , __func__,smp_processor_id (), vector); }else {
@@ -270,8 +259,6 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) } } -exiting_irq (); -set_irq_regs (old_regs); }
PATCH v9 25/39 x86/entry: Provide IDTENTRY_SYSVEC
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Provide IDTENTRY variants for system vectors to consolidate the different mechanisms to emit the ASM stubs for 32 an 64 bit.
On 64bit this also moves the stack switching from ASM to C code. 32bit will excute the system vectors w/o stack switching as before.
The simple variant is meant for "empty" system vectors like scheduler IPI and KVM posted interrupt vectors. These do not need the full glory of irq enter/exit handling with softirq processing and more.
---
V9: Use cond. RCU handling
--- arch/x86/entry/entry_32.S | 4 ++ arch/x86/entry/entry_64.S | 8 ++++ arch/x86/include/asm/idtentry.h | 79 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+)
arch/x86/entry/entry_32.S +4 -0
@@ -765,6 +765,10 @@ SYM_CODE_START_LOCAL(asm_\cfunc) SYM_CODE_END(asm_\cfunc) .endm +.macro idtentry_sysvec vector cfunc + idtentry \vector asm_\cfunc \cfunc has_error_code=0 +.endm + /* * Include the defines which emit the idt entries which are shared * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +8 -0
@@ -542,6 +542,14 @@ SYM_CODE_END(\asmsym) .endm /* + * System vectors which invoke their handlers directly and are not + * going through the regular common device interrupt handling code. + */ +.macro idtentry_sysvec vector cfunc + idtentry \vector asm_\cfunc \cfunc has_error_code=0 +.endm + +/* * MCE and DB exceptions */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +79 -0
97 lines not on this page · raw message
[tip: x86/entry] x86/entry: Provide IDTENTRY_SYSVEC
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 8c3d969a5352c3e68a42311852e20ee7530492c6 Gitweb: https://git.kernel.org/tip/8c3d969a5352c3e68a42311852e20ee7530492c6 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:38 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Provide IDTENTRY_SYSVEC
Provide IDTENTRY variants for system vectors to consolidate the different mechanisms to emit the ASM stubs for 32- and 64-bit.
On 64-bit this also moves the stack switching from ASM to C code. 32-bit will excute the system vectors w/o stack switching as before.
The simple variant is meant for "empty" system vectors like scheduler IPI and KVM posted interrupt vectors. These do not need the full glory of irq enter/exit handling with softirq processing and more.
--- arch/x86/entry/entry_32.S | 4 ++- arch/x86/entry/entry_64.S | 8 +++- arch/x86/include/asm/idtentry.h | 79 ++++++++++++++++++++++++++++++++- 3 files changed, 91 insertions(+)
arch/x86/entry/entry_32.S +4 -0
@@ -765,6 +765,10 @@ SYM_CODE_START_LOCAL(asm_\cfunc) SYM_CODE_END(asm_\cfunc) .endm +.macro idtentry_sysvec vector cfunc + idtentry \vector asm_\cfunc \cfunc has_error_code=0 +.endm + /* * Include the defines which emit the idt entries which are shared * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +8 -0
@@ -542,6 +542,14 @@ SYM_CODE_END(\asmsym) .endm /* + * System vectors which invoke their handlers directly and are not + * going through the regular common device interrupt handling code. + */ +.macro idtentry_sysvec vector cfunc + idtentry \vector asm_\cfunc \cfunc has_error_code=0 +.endm + +/* * MCE and DB exceptions */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +79 -0
97 lines not on this page · raw message
PATCH v9 26/39 x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Convert APIC interrupts to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
No functional change.
--- arch/x86/entry/entry_64.S | 6 ------ arch/x86/include/asm/entry_arch.h | 5 ----- arch/x86/include/asm/hw_irq.h | 4 ---- arch/x86/include/asm/idtentry.h | 8 ++++++++ arch/x86/include/asm/irq.h | 1 - arch/x86/include/asm/traps.h | 3 --- arch/x86/kernel/apic/apic.c | 23 +++++------------------ arch/x86/kernel/idt.c | 8 ++++---- arch/x86/kernel/irq.c | 5 ++--- 9 files changed, 19 insertions(+), 44 deletions(-)
arch/x86/entry/entry_64.S +0 -6
@@ -965,9 +965,6 @@ apicinterrupt3 REBOOT_VECTOR reboot_in apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt #endif -apicinterrupt LOCAL_TIMER_VECTOR apic_timer_interrupt smp_apic_timer_interrupt -apicinterrupt X86_PLATFORM_IPI_VECTOR x86_platform_ipi smp_x86_platform_ipi - #ifdef CONFIG_HAVE_KVM apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_posted_intr_wakeup_ipi
@@ -992,9 +989,6 @@ apicinterrupt CALL_FUNCTION_VECTOR call apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif -apicinterrupt ERROR_APIC_VECTOR error_interrupt smp_error_interrupt -apicinterrupt SPURIOUS_APIC_VECTOR spurious_apic_interrupt smp_spurious_apic_interrupt - #ifdef CONFIG_IRQ_WORK apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -5
@@ -33,11 +33,6 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_i*/ #ifdef CONFIG_X86_LOCAL_APIC -BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR) -BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR) -BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR) -BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR) -#ifdef CONFIG_IRQ_WORK BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR)#endif
arch/x86/include/asm/hw_irq.h +0 -4
@@ -29,16 +29,12 @@#include <asm/sections.h> /* Interrupt handlers registered during init_IRQ */ -extern asmlinkage void apic_timer_interrupt(void); -extern asmlinkage void x86_platform_ipi(void); extern asmlinkage void kvm_posted_intr_ipi(void); extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); extern asmlinkage void kvm_posted_intr_nested_ipi(void); -extern asmlinkage void error_interrupt(void); extern asmlinkage void irq_work_interrupt(void); extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +8 -0
@@ -565,6 +565,14 @@ DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER, comDECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , spurious_interrupt);#endif +/* System vector entry points */ +#ifdef CONFIG_X86_LOCAL_APIC +DECLARE_IDTENTRY_SYSVEC (ERROR_APIC_VECTOR , sysvec_error_interrupt); +DECLARE_IDTENTRY_SYSVEC (SPURIOUS_APIC_VECTOR , sysvec_spurious_apic_interrupt); +DECLARE_IDTENTRY_SYSVEC (LOCAL_TIMER_VECTOR , sysvec_apic_timer_interrupt); +DECLARE_IDTENTRY_SYSVEC (X86_PLATFORM_IPI_VECTOR , sysvec_x86_platform_ipi); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq.h +0 -1
@@ -46,7 +46,6 @@ extern void __init init_IRQ(void);void arch_trigger_cpumask_backtrace (const struct cpumask *mask,bool exclude_self); -extern __visible void smp_x86_platform_ipi(struct pt_regs *regs);#define arch_trigger_cpumask_backtrace arch_trigger_cpumask_backtrace#endif
arch/x86/include/asm/traps.h +0 -3
@@ -40,9 +40,6 @@ asmlinkage void smp_threshold_interrupt( asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);#endif -void smp_apic_timer_interrupt (struct pt_regs *regs); -void smp_error_interrupt (struct pt_regs *regs); -void smp_spurious_apic_interrupt (struct pt_regs *regs); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +5 -18
@@ -1121,23 +1121,14 @@ static void local_apic_timer_interrupt(v* [ if a single-CPU system runs an SMP kernel then we call the local * interrupt as well. Thus we cannot inline the local irq ... ] */ -__visible void __irq_entry smp_apic_timer_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_apic_timer_interrupt) {struct pt_regs *old_regs =set_irq_regs (regs); -/* -* NOTE! We'd better ACK the irq immediately, -* because timer handling can be slow. -* -* update_process_times() expects us to have done irq_enter(). -* Besides, if we don't timer interrupts ignore the global -* interrupt lock, which is the WrongThing (tm) to do. -*/ -entering_ack_irq (); +ack_APIC_irq ();trace_local_timer_entry (LOCAL_TIMER_VECTOR );local_apic_timer_interrupt ();trace_local_timer_exit (LOCAL_TIMER_VECTOR ); -exiting_irq ();set_irq_regs (old_regs); }
@@ -2162,7 +2153,7 @@ void __init register_lapic_address(unsig* trigger on an entry which is routed to the common_spurious idtentry * point. * -* Also called from smp_spurious_apic_interrupt(). +* Also called from sysvec_spurious_apic_interrupt(). */ DEFINE_IDTENTRY_IRQ (spurious_interrupt) {
@@ -2199,17 +2190,15 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)trace_spurious_apic_exit (vector); } -__visible void smp_spurious_apic_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_spurious_apic_interrupt) { -entering_irq ();__spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); -exiting_irq (); }/* * This interrupt should never happen with our APIC/SMP architecture */ -__visible void __irq_entry smp_error_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_error_interrupt) {static const char *const error_interrupt_reason[] = {"Send CS error" ,/* APIC Error Bit 0 */
@@ -2223,7 +2212,6 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt) };u32 v, i =0 ; -entering_irq ();trace_error_apic_entry (ERROR_APIC_VECTOR );/* First tickle the hardware, only then report what went on. -- REW */
@@ -2247,7 +2235,6 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)apic_printk (APIC_DEBUG ,KERN_CONT "\n" );trace_error_apic_exit (ERROR_APIC_VECTOR ); -exiting_irq (); }/**
arch/x86/kernel/idt.c +4 -4
@@ -132,8 +132,8 @@ static const __initconst struct idt_data#endif #ifdef CONFIG_X86_LOCAL_APIC - INTG(LOCAL_TIMER_VECTOR, apic_timer_interrupt), - INTG(X86_PLATFORM_IPI_VECTOR, x86_platform_ipi), + INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), + INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi),# ifdef CONFIG_HAVE_KVM INTG (POSTED_INTR_VECTOR , kvm_posted_intr_ipi), INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi),
@@ -145,8 +145,8 @@ static const __initconst struct idt_data #ifdef CONFIG_X86_UV INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), #endif - INTG(SPURIOUS_APIC_VECTOR, spurious_apic_interrupt), - INTG(ERROR_APIC_VECTOR, error_interrupt), + INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), + INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), #endif };
arch/x86/kernel/irq.c +2 -3
@@ -268,17 +268,16 @@ void (*x86_platform_ipi_callback)(void)/* * Handler for X86_PLATFORM_IPI_VECTOR. */ -__visible void __irq_entry smp_x86_platform_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_x86_platform_ipi) {struct pt_regs *old_regs =set_irq_regs (regs); -entering_ack_irq (); +ack_APIC_irq ();trace_x86_platform_ipi_entry (X86_PLATFORM_IPI_VECTOR );inc_irq_stat (x86_platform_ipis);if (x86_platform_ipi_callback)x86_platform_ipi_callback ();trace_x86_platform_ipi_exit (X86_PLATFORM_IPI_VECTOR ); -exiting_irq ();set_irq_regs (old_regs); }#endif
[tip: x86/entry] x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: d1fa4e0e06cfcc3284f6bf51f39ae4a34f3fe434 Gitweb: https://git.kernel.org/tip/d1fa4e0e06cfcc3284f6bf51f39ae4a34f3fe434 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:39 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC
Convert APIC interrupts to IDTENTRY_SYSVEC:
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64-bit
- Remove the BUILD_INTERRUPT entries in 32-bit
- Remove the old prototypes
No functional change.
--- arch/x86/entry/entry_64.S | 6 ------ arch/x86/include/asm/entry_arch.h | 5 ----- arch/x86/include/asm/hw_irq.h | 4 ---- arch/x86/include/asm/idtentry.h | 8 ++++++++ arch/x86/include/asm/irq.h | 1 - arch/x86/include/asm/traps.h | 3 --- arch/x86/kernel/apic/apic.c | 23 +++++------------------ arch/x86/kernel/idt.c | 8 ++++---- arch/x86/kernel/irq.c | 5 ++--- 9 files changed, 19 insertions(+), 44 deletions(-)
arch/x86/entry/entry_64.S +0 -6
@@ -965,9 +965,6 @@ apicinterrupt3 REBOOT_VECTOR reboot_interrupt smp_reboot_interrupt apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt #endif -apicinterrupt LOCAL_TIMER_VECTOR apic_timer_interrupt smp_apic_timer_interrupt -apicinterrupt X86_PLATFORM_IPI_VECTOR x86_platform_ipi smp_x86_platform_ipi - #ifdef CONFIG_HAVE_KVM apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_posted_intr_wakeup_ipi
@@ -992,9 +989,6 @@ apicinterrupt CALL_FUNCTION_VECTOR call_function_interrupt smp_call_function_i apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif -apicinterrupt ERROR_APIC_VECTOR error_interrupt smp_error_interrupt -apicinterrupt SPURIOUS_APIC_VECTOR spurious_apic_interrupt smp_spurious_apic_interrupt - #ifdef CONFIG_IRQ_WORK apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -5
@@ -33,11 +33,6 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)*/ #ifdef CONFIG_X86_LOCAL_APIC -BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR) -BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR) -BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR) -BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR) -#ifdef CONFIG_IRQ_WORK BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR)#endif
arch/x86/include/asm/hw_irq.h +0 -4
@@ -29,16 +29,12 @@#include <asm/sections.h> /* Interrupt handlers registered during init_IRQ */ -extern asmlinkage void apic_timer_interrupt(void); -extern asmlinkage void x86_platform_ipi(void); extern asmlinkage void kvm_posted_intr_ipi(void); extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); extern asmlinkage void kvm_posted_intr_nested_ipi(void); -extern asmlinkage void error_interrupt(void); extern asmlinkage void irq_work_interrupt(void); extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +8 -0
@@ -569,6 +569,14 @@ DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER, common_interrupt);DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , spurious_interrupt);#endif +/* System vector entry points */ +#ifdef CONFIG_X86_LOCAL_APIC +DECLARE_IDTENTRY_SYSVEC (ERROR_APIC_VECTOR , sysvec_error_interrupt); +DECLARE_IDTENTRY_SYSVEC (SPURIOUS_APIC_VECTOR , sysvec_spurious_apic_interrupt); +DECLARE_IDTENTRY_SYSVEC (LOCAL_TIMER_VECTOR , sysvec_apic_timer_interrupt); +DECLARE_IDTENTRY_SYSVEC (X86_PLATFORM_IPI_VECTOR , sysvec_x86_platform_ipi); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq.h +0 -1
@@ -46,7 +46,6 @@ extern void __init init_IRQ(void);void arch_trigger_cpumask_backtrace (const struct cpumask *mask,bool exclude_self); -extern __visible void smp_x86_platform_ipi(struct pt_regs *regs);#define arch_trigger_cpumask_backtrace arch_trigger_cpumask_backtrace#endif
arch/x86/include/asm/traps.h +0 -3
@@ -40,9 +40,6 @@ asmlinkage void smp_threshold_interrupt(struct pt_regs *regs); asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);#endif -void smp_apic_timer_interrupt (struct pt_regs *regs); -void smp_error_interrupt (struct pt_regs *regs); -void smp_spurious_apic_interrupt (struct pt_regs *regs); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +5 -18
@@ -1121,23 +1121,14 @@ static void local_apic_timer_interrupt(void)* [ if a single-CPU system runs an SMP kernel then we call the local * interrupt as well. Thus we cannot inline the local irq ... ] */ -__visible void __irq_entry smp_apic_timer_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_apic_timer_interrupt) {struct pt_regs *old_regs =set_irq_regs (regs); -/* -* NOTE! We'd better ACK the irq immediately, -* because timer handling can be slow. -* -* update_process_times() expects us to have done irq_enter(). -* Besides, if we don't timer interrupts ignore the global -* interrupt lock, which is the WrongThing (tm) to do. -*/ -entering_ack_irq (); +ack_APIC_irq ();trace_local_timer_entry (LOCAL_TIMER_VECTOR );local_apic_timer_interrupt ();trace_local_timer_exit (LOCAL_TIMER_VECTOR ); -exiting_irq ();set_irq_regs (old_regs); }
@@ -2162,7 +2153,7 @@ void __init register_lapic_address(unsigned long address)* trigger on an entry which is routed to the common_spurious idtentry * point. * -* Also called from smp_spurious_apic_interrupt(). +* Also called from sysvec_spurious_apic_interrupt(). */ DEFINE_IDTENTRY_IRQ (spurious_interrupt) {
@@ -2199,17 +2190,15 @@ out:trace_spurious_apic_exit (vector); } -__visible void smp_spurious_apic_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_spurious_apic_interrupt) { -entering_irq ();__spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); -exiting_irq (); }/* * This interrupt should never happen with our APIC/SMP architecture */ -__visible void __irq_entry smp_error_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_error_interrupt) {static const char *const error_interrupt_reason[] = {"Send CS error" ,/* APIC Error Bit 0 */
@@ -2223,7 +2212,6 @@ __visible void __irq_entry smp_error_interrupt(struct pt_regs *regs) };u32 v, i =0 ; -entering_irq ();trace_error_apic_entry (ERROR_APIC_VECTOR );/* First tickle the hardware, only then report what went on. -- REW */
@@ -2247,7 +2235,6 @@ __visible void __irq_entry smp_error_interrupt(struct pt_regs *regs)apic_printk (APIC_DEBUG ,KERN_CONT "\n" );trace_error_apic_exit (ERROR_APIC_VECTOR ); -exiting_irq (); }/**
arch/x86/kernel/idt.c +4 -4
@@ -132,8 +132,8 @@ static const __initconst struct idt_data apic_idts[] = {#endif #ifdef CONFIG_X86_LOCAL_APIC - INTG(LOCAL_TIMER_VECTOR, apic_timer_interrupt), - INTG(X86_PLATFORM_IPI_VECTOR, x86_platform_ipi), + INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), + INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi),# ifdef CONFIG_HAVE_KVM INTG (POSTED_INTR_VECTOR , kvm_posted_intr_ipi), INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi),
@@ -145,8 +145,8 @@ static const __initconst struct idt_data apic_idts[] = { #ifdef CONFIG_X86_UV INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), #endif - INTG(SPURIOUS_APIC_VECTOR, spurious_apic_interrupt), - INTG(ERROR_APIC_VECTOR, error_interrupt), + INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), + INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), #endif };
arch/x86/kernel/irq.c +2 -3
@@ -268,17 +268,16 @@ void (*x86_platform_ipi_callback)(void) = NULL;/* * Handler for X86_PLATFORM_IPI_VECTOR. */ -__visible void __irq_entry smp_x86_platform_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_x86_platform_ipi) {struct pt_regs *old_regs =set_irq_regs (regs); -entering_ack_irq (); +ack_APIC_irq ();trace_x86_platform_ipi_entry (X86_PLATFORM_IPI_VECTOR );inc_irq_stat (x86_platform_ipis);if (x86_platform_ipi_callback)x86_platform_ipi_callback ();trace_x86_platform_ipi_exit (X86_PLATFORM_IPI_VECTOR ); -exiting_irq ();set_irq_regs (old_regs); }#endif
PATCH v9 27/39 x86/entry: Convert SMP system vectors to IDTENTRY_SYSVEC
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Convert SMP system vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
No functional change.
--- arch/x86/entry/entry_64.S | 7 ------- arch/x86/include/asm/entry_arch.h | 4 ---- arch/x86/include/asm/hw_irq.h | 5 ----- arch/x86/include/asm/idtentry.h | 7 +++++++ arch/x86/include/asm/traps.h | 2 -- arch/x86/kernel/apic/vector.c | 5 ++--- arch/x86/kernel/idt.c | 10 +++++----- arch/x86/kernel/smp.c | 18 +++++++----------- 8 files changed, 21 insertions(+), 37 deletions(-)
arch/x86/entry/entry_64.S +0 -7
@@ -956,11 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm -#ifdef CONFIG_SMP -apicinterrupt3 IRQ_MOVE_CLEANUP_VECTOR irq_move_cleanup_interrupt smp_irq_move_cleanup_interrupt -apicinterrupt3 REBOOT_VECTOR reboot_interrupt smp_reboot_interrupt -#endif - #ifdef CONFIG_X86_UV apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt #endif
@@ -984,8 +979,6 @@ apicinterrupt THERMAL_APIC_VECTOR therm #endif #ifdef CONFIG_SMP -apicinterrupt CALL_FUNCTION_SINGLE_VECTOR call_function_single_interrupt smp_call_function_single_interrupt -apicinterrupt CALL_FUNCTION_VECTOR call_function_interrupt smp_call_function_interrupt apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -4
@@ -12,10 +12,6 @@*/ #ifdef CONFIG_SMP BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR) -BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR) -BUILD_INTERRUPT(call_function_single_interrupt,CALL_FUNCTION_SINGLE_VECTOR) -BUILD_INTERRUPT(irq_move_cleanup_interrupt, IRQ_MOVE_CLEANUP_VECTOR) -BUILD_INTERRUPT(reboot_interrupt, REBOOT_VECTOR)#endif #ifdef CONFIG_HAVE_KVM
arch/x86/include/asm/hw_irq.h +0 -5
@@ -38,14 +38,9 @@ extern asmlinkage void uv_bau_message_in extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void); -extern asmlinkage void irq_move_cleanup_interrupt(void); -extern asmlinkage void reboot_interrupt(void); extern asmlinkage void threshold_interrupt(void); extern asmlinkage void deferred_error_interrupt(void); -extern asmlinkage void call_function_interrupt(void); -extern asmlinkage void call_function_single_interrupt(void); -#ifdef CONFIG_X86_LOCAL_APIC struct irq_data ;struct pci_dev ;
arch/x86/include/asm/idtentry.h +7 -0
@@ -573,6 +573,13 @@ DECLARE_IDTENTRY_SYSVEC(LOCAL_TIMER_VECTDECLARE_IDTENTRY_SYSVEC (X86_PLATFORM_IPI_VECTOR , sysvec_x86_platform_ipi);#endif +#ifdef CONFIG_SMP +DECLARE_IDTENTRY_SYSVEC (IRQ_MOVE_CLEANUP_VECTOR , sysvec_irq_move_cleanup); +DECLARE_IDTENTRY_SYSVEC (REBOOT_VECTOR , sysvec_reboot); +DECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_SINGLE_VECTOR , sysvec_call_function_single); +DECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_VECTOR , sysvec_call_function); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/traps.h +0 -2
@@ -40,8 +40,6 @@ asmlinkage void smp_threshold_interrupt( asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);#endif -asmlinkage void smp_irq_move_cleanup_interrupt(void); -#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message, struct pt_regs *regs,
arch/x86/kernel/apic/vector.c +2 -3
@@ -861,13 +861,13 @@ static void free_moved_vector(struct api apicd->move_in_progress =0 ; } -asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void) +DEFINE_IDTENTRY_SYSVEC (sysvec_irq_move_cleanup) {struct hlist_head *clhead =this_cpu_ptr (&cleanup_list);struct apic_chip_data *apicd;struct hlist_node *tmp; -entering_ack_irq (); +ack_APIC_irq ();/* Prevent vectors vanishing under us */ raw_spin_lock (&vector_lock);
@@ -892,7 +892,6 @@ asmlinkage __visible void __irq_entry sm }raw_spin_unlock (&vector_lock); -exiting_irq (); }static void __send_cleanup_vector (struct apic_chip_data *apicd)
arch/x86/kernel/idt.c +5 -5
@@ -112,11 +112,11 @@ static const __initconst struct idt_data*/ static const __initconst struct idt_data apic_idts[] = { #ifdef CONFIG_SMP -INTG (RESCHEDULE_VECTOR , reschedule_interrupt), -INTG (CALL_FUNCTION_VECTOR , call_function_interrupt), -INTG (CALL_FUNCTION_SINGLE_VECTOR , call_function_single_interrupt), -INTG (IRQ_MOVE_CLEANUP_VECTOR , irq_move_cleanup_interrupt), -INTG (REBOOT_VECTOR , reboot_interrupt), +INTG (RESCHEDULE_VECTOR , reschedule_interrupt), +INTG (CALL_FUNCTION_VECTOR , asm_sysvec_call_function), +INTG (CALL_FUNCTION_SINGLE_VECTOR , asm_sysvec_call_function_single), +INTG (IRQ_MOVE_CLEANUP_VECTOR , asm_sysvec_irq_move_cleanup), +INTG (REBOOT_VECTOR , asm_sysvec_reboot), #endif #ifdef CONFIG_X86_THERMAL_VECTOR
arch/x86/kernel/smp.c +7 -11
@@ -27,6 +27,7 @@#include <asm/mmu_context.h> #include <asm/proto.h> #include <asm/apic.h> +#include <asm/idtentry.h> #include <asm/nmi.h> #include <asm/mce.h> #include <asm/trace/irq_vectors.h>
@@ -130,13 +131,11 @@ static int smp_stop_nmi_callback(unsigne/* * this function calls the 'stop' function on all other CPUs in the system. */ - -asmlinkage __visible void smp_reboot_interrupt(void) +DEFINE_IDTENTRY_SYSVEC (sysvec_reboot) { -ipi_entering_ack_irq (); +ack_APIC_irq ();cpu_emergency_vmxoff ();stop_this_cpu (NULL ); -irq_exit (); }static int register_stop_handler (void )
@@ -227,7 +226,6 @@ static void native_stop_other_cpus(int w {ack_APIC_irq ();inc_irq_stat (irq_resched_count); -kvm_set_cpu_l1tf_flush_l1d ();if (trace_resched_ipi_enabled ()) {/*
@@ -244,24 +242,22 @@ static void native_stop_other_cpus(int wscheduler_ipi (); } -__visible void __irq_entry smp_call_function_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_call_function) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_call_function_entry (CALL_FUNCTION_VECTOR );inc_irq_stat (irq_call_count);generic_smp_call_function_interrupt ();trace_call_function_exit (CALL_FUNCTION_VECTOR ); -exiting_irq (); } -__visible void __irq_entry smp_call_function_single_interrupt(struct pt_regs *r) +DEFINE_IDTENTRY_SYSVEC (sysvec_call_function_single) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_call_function_single_entry (CALL_FUNCTION_SINGLE_VECTOR );inc_irq_stat (irq_call_count);generic_smp_call_function_single_interrupt ();trace_call_function_single_exit (CALL_FUNCTION_SINGLE_VECTOR ); -exiting_irq (); }static int __init nonmi_ipi_setup (char *str)
[tip: x86/entry] x86/entry: Convert SMP system vectors to IDTENTRY_SYSVEC
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 8f2d02673efc50ad7075fbbfce465766cf8f1a20 Gitweb: https://git.kernel.org/tip/8f2d02673efc50ad7075fbbfce465766cf8f1a20 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:40 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Convert SMP system vectors to IDTENTRY_SYSVEC
Convert SMP system vectors to IDTENTRY_SYSVEC:
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64-bit
- Remove the BUILD_INTERRUPT entries in 32-bit
- Remove the old prototypes
No functional change.
--- arch/x86/entry/entry_64.S | 7 ------- arch/x86/include/asm/entry_arch.h | 4 ---- arch/x86/include/asm/hw_irq.h | 5 ----- arch/x86/include/asm/idtentry.h | 7 +++++++ arch/x86/include/asm/traps.h | 2 -- arch/x86/kernel/apic/vector.c | 5 ++--- arch/x86/kernel/idt.c | 10 +++++----- arch/x86/kernel/smp.c | 18 +++++++----------- 8 files changed, 21 insertions(+), 37 deletions(-)
arch/x86/entry/entry_64.S +0 -7
@@ -956,11 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm -#ifdef CONFIG_SMP -apicinterrupt3 IRQ_MOVE_CLEANUP_VECTOR irq_move_cleanup_interrupt smp_irq_move_cleanup_interrupt -apicinterrupt3 REBOOT_VECTOR reboot_interrupt smp_reboot_interrupt -#endif - #ifdef CONFIG_X86_UV apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt #endif
@@ -984,8 +979,6 @@ apicinterrupt THERMAL_APIC_VECTOR thermal_interrupt smp_thermal_interrupt #endif #ifdef CONFIG_SMP -apicinterrupt CALL_FUNCTION_SINGLE_VECTOR call_function_single_interrupt smp_call_function_single_interrupt -apicinterrupt CALL_FUNCTION_VECTOR call_function_interrupt smp_call_function_interrupt apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -4
@@ -12,10 +12,6 @@*/ #ifdef CONFIG_SMP BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR) -BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR) -BUILD_INTERRUPT(call_function_single_interrupt,CALL_FUNCTION_SINGLE_VECTOR) -BUILD_INTERRUPT(irq_move_cleanup_interrupt, IRQ_MOVE_CLEANUP_VECTOR) -BUILD_INTERRUPT(reboot_interrupt, REBOOT_VECTOR)#endif #ifdef CONFIG_HAVE_KVM
arch/x86/include/asm/hw_irq.h +0 -5
@@ -38,14 +38,9 @@ extern asmlinkage void uv_bau_message_intr1(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void); -extern asmlinkage void irq_move_cleanup_interrupt(void); -extern asmlinkage void reboot_interrupt(void); extern asmlinkage void threshold_interrupt(void); extern asmlinkage void deferred_error_interrupt(void); -extern asmlinkage void call_function_interrupt(void); -extern asmlinkage void call_function_single_interrupt(void); -#ifdef CONFIG_X86_LOCAL_APIC struct irq_data ;struct pci_dev ;
arch/x86/include/asm/idtentry.h +7 -0
@@ -577,6 +577,13 @@ DECLARE_IDTENTRY_SYSVEC(LOCAL_TIMER_VECTOR, sysvec_apic_timer_interrupt);DECLARE_IDTENTRY_SYSVEC (X86_PLATFORM_IPI_VECTOR , sysvec_x86_platform_ipi);#endif +#ifdef CONFIG_SMP +DECLARE_IDTENTRY_SYSVEC (IRQ_MOVE_CLEANUP_VECTOR , sysvec_irq_move_cleanup); +DECLARE_IDTENTRY_SYSVEC (REBOOT_VECTOR , sysvec_reboot); +DECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_SINGLE_VECTOR , sysvec_call_function_single); +DECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_VECTOR , sysvec_call_function); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/traps.h +0 -2
@@ -40,8 +40,6 @@ asmlinkage void smp_threshold_interrupt(struct pt_regs *regs); asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);#endif -asmlinkage void smp_irq_move_cleanup_interrupt(void); -#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message, struct pt_regs *regs,
arch/x86/kernel/apic/vector.c +2 -3
@@ -861,13 +861,13 @@ static void free_moved_vector(struct apic_chip_data *apicd) apicd->move_in_progress =0 ; } -asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void) +DEFINE_IDTENTRY_SYSVEC (sysvec_irq_move_cleanup) {struct hlist_head *clhead =this_cpu_ptr (&cleanup_list);struct apic_chip_data *apicd;struct hlist_node *tmp; -entering_ack_irq (); +ack_APIC_irq ();/* Prevent vectors vanishing under us */ raw_spin_lock (&vector_lock);
@@ -892,7 +892,6 @@ asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void) }raw_spin_unlock (&vector_lock); -exiting_irq (); }static void __send_cleanup_vector (struct apic_chip_data *apicd)
arch/x86/kernel/idt.c +5 -5
@@ -112,11 +112,11 @@ static const __initconst struct idt_data def_idts[] = {*/ static const __initconst struct idt_data apic_idts[] = { #ifdef CONFIG_SMP -INTG (RESCHEDULE_VECTOR , reschedule_interrupt), -INTG (CALL_FUNCTION_VECTOR , call_function_interrupt), -INTG (CALL_FUNCTION_SINGLE_VECTOR , call_function_single_interrupt), -INTG (IRQ_MOVE_CLEANUP_VECTOR , irq_move_cleanup_interrupt), -INTG (REBOOT_VECTOR , reboot_interrupt), +INTG (RESCHEDULE_VECTOR , reschedule_interrupt), +INTG (CALL_FUNCTION_VECTOR , asm_sysvec_call_function), +INTG (CALL_FUNCTION_SINGLE_VECTOR , asm_sysvec_call_function_single), +INTG (IRQ_MOVE_CLEANUP_VECTOR , asm_sysvec_irq_move_cleanup), +INTG (REBOOT_VECTOR , asm_sysvec_reboot), #endif #ifdef CONFIG_X86_THERMAL_VECTOR
arch/x86/kernel/smp.c +7 -11
@@ -27,6 +27,7 @@#include <asm/mmu_context.h> #include <asm/proto.h> #include <asm/apic.h> +#include <asm/idtentry.h> #include <asm/nmi.h> #include <asm/mce.h> #include <asm/trace/irq_vectors.h>
@@ -130,13 +131,11 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs)/* * this function calls the 'stop' function on all other CPUs in the system. */ - -asmlinkage __visible void smp_reboot_interrupt(void) +DEFINE_IDTENTRY_SYSVEC (sysvec_reboot) { -ipi_entering_ack_irq (); +ack_APIC_irq ();cpu_emergency_vmxoff ();stop_this_cpu (NULL ); -irq_exit (); }static int register_stop_handler (void )
@@ -227,7 +226,6 @@ __visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs) {ack_APIC_irq ();inc_irq_stat (irq_resched_count); -kvm_set_cpu_l1tf_flush_l1d ();if (trace_resched_ipi_enabled ()) {/*
@@ -244,24 +242,22 @@ __visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs)scheduler_ipi (); } -__visible void __irq_entry smp_call_function_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_call_function) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_call_function_entry (CALL_FUNCTION_VECTOR );inc_irq_stat (irq_call_count);generic_smp_call_function_interrupt ();trace_call_function_exit (CALL_FUNCTION_VECTOR ); -exiting_irq (); } -__visible void __irq_entry smp_call_function_single_interrupt(struct pt_regs *r) +DEFINE_IDTENTRY_SYSVEC (sysvec_call_function_single) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_call_function_single_entry (CALL_FUNCTION_SINGLE_VECTOR );inc_irq_stat (irq_call_count);generic_smp_call_function_single_interrupt ();trace_call_function_single_exit (CALL_FUNCTION_SINGLE_VECTOR ); -exiting_irq (); }static int __init nonmi_ipi_setup (char *str)
PATCH v9 28/39 x86/entry: Convert various system vectors
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Convert various system vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
No functional change.
--- arch/x86/entry/entry_64.S | 19 ------------------- arch/x86/include/asm/apic.h | 13 ------------- arch/x86/include/asm/entry_arch.h | 25 ------------------------- arch/x86/include/asm/hw_irq.h | 6 ------ arch/x86/include/asm/idtentry.h | 22 ++++++++++++++++++++++ arch/x86/include/asm/irq_work.h | 1 - arch/x86/include/asm/traps.h | 5 ----- arch/x86/include/asm/uv/uv_bau.h | 8 ++------ arch/x86/kernel/cpu/mce/amd.c | 5 ++--- arch/x86/kernel/cpu/mce/therm_throt.c | 5 ++--- arch/x86/kernel/cpu/mce/threshold.c | 5 ++--- arch/x86/kernel/idt.c | 28 ++++++++++++++-------------- arch/x86/kernel/irq_work.c | 6 +++--- arch/x86/platform/uv/tlb_uv.c | 2 +- 14 files changed, 48 insertions(+), 102 deletions(-)
arch/x86/entry/entry_64.S +0 -19
@@ -956,9 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm -#ifdef CONFIG_X86_UV -apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt -#endif #ifdef CONFIG_HAVE_KVM apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi
@@ -966,26 +963,10 @@ apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR apicinterrupt3 POSTED_INTR_NESTED_VECTOR kvm_posted_intr_nested_ipi smp_kvm_posted_intr_nested_ipi #endif -#ifdef CONFIG_X86_MCE_THRESHOLD -apicinterrupt THRESHOLD_APIC_VECTOR threshold_interrupt smp_threshold_interrupt -#endif - -#ifdef CONFIG_X86_MCE_AMD -apicinterrupt DEFERRED_ERROR_VECTOR deferred_error_interrupt smp_deferred_error_interrupt -#endif - -#ifdef CONFIG_X86_THERMAL_VECTOR -apicinterrupt THERMAL_APIC_VECTOR thermal_interrupt smp_thermal_interrupt -#endif - #ifdef CONFIG_SMP apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif -#ifdef CONFIG_IRQ_WORK -apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt -#endif - /* * Reload gs selector with exception handling * edi: new selector
arch/x86/include/asm/apic.h +0 -13
@@ -534,24 +534,11 @@ static inline void entering_ack_irq(void ack_APIC_irq(); } -static inline void ipi_entering_ack_irq (void ) -{ -irq_enter (); -ack_APIC_irq (); -kvm_set_cpu_l1tf_flush_l1d (); -} -static inline void exiting_irq (void ) {irq_exit (); } -static inline void exiting_ack_irq (void ) -{ -ack_APIC_irq (); -irq_exit (); -} -extern void ioapic_zap_locks (void );#endif /* _ASM_X86_APIC_H */
arch/x86/include/asm/entry_arch.h +0 -25
@@ -20,28 +20,3 @@ BUILD_INTERRUPT(kvm_posted_intr_wakeup_i BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)#endif -/* -* every pentium local APIC has two 'local interrupts', with a -* soft-definable vector attached to both interrupts, one of -* which is a timer interrupt, the other one is error counter -* overflow. Linux uses the local APIC timer interrupt to get -* a much simpler SMP time architecture: -*/ -#ifdef CONFIG_X86_LOCAL_APIC - -#ifdef CONFIG_IRQ_WORK -BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR) -#endif - -#ifdef CONFIG_X86_THERMAL_VECTOR -BUILD_INTERRUPT(thermal_interrupt,THERMAL_APIC_VECTOR) -#endif - -#ifdef CONFIG_X86_MCE_THRESHOLD -BUILD_INTERRUPT(threshold_interrupt,THRESHOLD_APIC_VECTOR) -#endif - -#ifdef CONFIG_X86_MCE_AMD -BUILD_INTERRUPT(deferred_error_interrupt, DEFERRED_ERROR_VECTOR) -#endif -#endif
arch/x86/include/asm/hw_irq.h +0 -6
@@ -32,15 +32,9 @@ extern asmlinkage void kvm_posted_intr_ipi(void); extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); extern asmlinkage void kvm_posted_intr_nested_ipi(void); -extern asmlinkage void irq_work_interrupt(void); -extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void); -extern asmlinkage void threshold_interrupt(void); -extern asmlinkage void deferred_error_interrupt(void); -#ifdef CONFIG_X86_LOCAL_APIC struct irq_data ;struct pci_dev ;
arch/x86/include/asm/idtentry.h +22 -0
@@ -580,6 +580,28 @@ DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SIDECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_VECTOR , sysvec_call_function);#endif +#ifdef CONFIG_X86_LOCAL_APIC +# ifdef CONFIG_X86_UV +DECLARE_IDTENTRY_SYSVEC (UV_BAU_MESSAGE , sysvec_uv_bau_message); +# endif + +# ifdef CONFIG_X86_MCE_THRESHOLD +DECLARE_IDTENTRY_SYSVEC (THRESHOLD_APIC_VECTOR , sysvec_threshold); +# endif + +# ifdef CONFIG_X86_MCE_AMD +DECLARE_IDTENTRY_SYSVEC (DEFERRED_ERROR_VECTOR , sysvec_deferred_error); +# endif + +# ifdef CONFIG_X86_THERMAL_VECTOR +DECLARE_IDTENTRY_SYSVEC (THERMAL_APIC_VECTOR , sysvec_thermal); +# endif + +# ifdef CONFIG_IRQ_WORK +DECLARE_IDTENTRY_SYSVEC (IRQ_WORK_VECTOR , sysvec_irq_work); +# endif +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq_work.h +0 -1
@@ -10,7 +10,6 @@ static inline bool arch_irq_work_has_int return boot_cpu_has(X86_FEATURE_APIC); }extern void arch_irq_work_raise (void ); -extern __visible void smp_irq_work_interrupt(struct pt_regs *regs);#else static inline bool arch_irq_work_has_interrupt (void ) {
arch/x86/include/asm/traps.h +0 -5
@@ -34,11 +34,6 @@ static inline int get_si_code(unsigned lextern int panic_on_unrecovered_nmi;void math_emulate (struct math_emu_info *); -#ifndef CONFIG_X86_32 -asmlinkage void smp_thermal_interrupt(struct pt_regs *regs); -asmlinkage void smp_threshold_interrupt(struct pt_regs *regs); -asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs); -#endif #ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message,
arch/x86/include/asm/uv/uv_bau.h +2 -6
@@ -12,6 +12,8 @@#define _ASM_X86_UV_UV_BAU_H#include <linux/bitmap.h> +#include <asm/idtentry.h> +#define BITSPERBYTE 8/*
@@ -799,12 +801,6 @@ static inline void bau_cpubits_clear(strbitmap_zero (&dstp->bits, nbits); } -extern void uv_bau_message_intr1 (void ); -#ifdef CONFIG_TRACING -#define trace_uv_bau_message_intr1 uv_bau_message_intr1 -#endif -extern void uv_bau_timeout_intr1 (void ); -struct atomic_short {short counter; };
arch/x86/kernel/cpu/mce/amd.c +2 -3
@@ -907,14 +907,13 @@ static void __log_error(unsigned int banmce_log (&m); } -asmlinkage __visible void __irq_entry smp_deferred_error_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_deferred_error) { -entering_irq ();trace_deferred_error_apic_entry (DEFERRED_ERROR_VECTOR );inc_irq_stat (irq_deferred_error_count);deferred_error_int_vector ();trace_deferred_error_apic_exit (DEFERRED_ERROR_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }/*
arch/x86/kernel/cpu/mce/therm_throt.c +2 -3
@@ -614,14 +614,13 @@ static void unexpected_thermal_interruptstatic void (*smp_thermal_vector)(void ) = unexpected_thermal_interrupt; -asmlinkage __visible void __irq_entry smp_thermal_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_thermal) { -entering_irq ();trace_thermal_apic_entry (THERMAL_APIC_VECTOR );inc_irq_stat (irq_thermal_count);smp_thermal_vector ();trace_thermal_apic_exit (THERMAL_APIC_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }/* Thermal monitoring depends on APIC, ACPI and clock modulation */
arch/x86/kernel/cpu/mce/threshold.c +2 -3
@@ -21,12 +21,11 @@ static void default_threshold_interrupt(void (*mce_threshold_vector)(void ) = default_threshold_interrupt; -asmlinkage __visible void __irq_entry smp_threshold_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_threshold) { -entering_irq ();trace_threshold_apic_entry (THRESHOLD_APIC_VECTOR );inc_irq_stat (irq_threshold_count);mce_threshold_vector ();trace_threshold_apic_exit (THRESHOLD_APIC_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }
arch/x86/kernel/idt.c +14 -14
@@ -120,33 +120,33 @@ static const __initconst struct idt_data#endif #ifdef CONFIG_X86_THERMAL_VECTOR - INTG(THERMAL_APIC_VECTOR, thermal_interrupt), + INTG(THERMAL_APIC_VECTOR, asm_sysvec_thermal),#endif #ifdef CONFIG_X86_MCE_THRESHOLD - INTG(THRESHOLD_APIC_VECTOR, threshold_interrupt), + INTG(THRESHOLD_APIC_VECTOR, asm_sysvec_threshold),#endif #ifdef CONFIG_X86_MCE_AMD - INTG(DEFERRED_ERROR_VECTOR, deferred_error_interrupt), + INTG(DEFERRED_ERROR_VECTOR, asm_sysvec_deferred_error),#endif #ifdef CONFIG_X86_LOCAL_APIC - INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), - INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi), + INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), + INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi), # ifdef CONFIG_HAVE_KVM - INTG(POSTED_INTR_VECTOR, kvm_posted_intr_ipi), - INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi), - INTG(POSTED_INTR_NESTED_VECTOR, kvm_posted_intr_nested_ipi), + INTG(POSTED_INTR_VECTOR, kvm_posted_intr_ipi), + INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi), + INTG(POSTED_INTR_NESTED_VECTOR, kvm_posted_intr_nested_ipi), # endif # ifdef CONFIG_IRQ_WORK - INTG(IRQ_WORK_VECTOR, irq_work_interrupt), + INTG(IRQ_WORK_VECTOR, asm_sysvec_irq_work), # endif -#ifdef CONFIG_X86_UV - INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), -#endif - INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), - INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), +# ifdef CONFIG_X86_UV + INTG(UV_BAU_MESSAGE, asm_sysvec_uv_bau_message), +# endif + INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), + INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), #endif };
arch/x86/kernel/irq_work.c +3 -3
@@ -9,18 +9,18 @@#include <linux/irq_work.h> #include <linux/hardirq.h> #include <asm/apic.h> +#include <asm/idtentry.h> #include <asm/trace/irq_vectors.h> #include <linux/interrupt.h> #ifdef CONFIG_X86_LOCAL_APIC -__visible void __irq_entry smp_irq_work_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_irq_work) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_irq_work_entry (IRQ_WORK_VECTOR );inc_irq_stat (apic_irq_work_irqs);irq_work_run ();trace_irq_work_exit (IRQ_WORK_VECTOR ); -exiting_irq (); }void arch_irq_work_raise (void )
arch/x86/platform/uv/tlb_uv.c +1 -1
@@ -1272,7 +1272,7 @@ static void process_uv2_message(struct m* (the resource will not be freed until noninterruptable cpus see this * interrupt; hardware may timeout the s/w ack and reply ERROR) */ -void uv_bau_message_interrupt (struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_uv_bau_message) {int count =0 ;cycles_t time_start;
[tip: x86/entry] x86/entry: Convert various system vectors
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 41a5e11a9f35e97cc8872ffdb379319ea584f997 Gitweb: https://git.kernel.org/tip/41a5e11a9f35e97cc8872ffdb379319ea584f997 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:41 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Convert various system vectors
Convert various system vectors to IDTENTRY_SYSVEC:
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64-bit
- Remove the BUILD_INTERRUPT entries in 32-bit
- Remove the old prototypes
No functional change.
--- arch/x86/entry/entry_64.S | 19 +------------------ arch/x86/include/asm/apic.h | 13 +------------ arch/x86/include/asm/entry_arch.h | 25 +----------------------- arch/x86/include/asm/hw_irq.h | 6 +------ arch/x86/include/asm/idtentry.h | 22 ++++++++++++++++++++- arch/x86/include/asm/irq_work.h | 1 +- arch/x86/include/asm/traps.h | 5 +----- arch/x86/include/asm/uv/uv_bau.h | 8 +------ arch/x86/kernel/cpu/mce/amd.c | 5 ++--- arch/x86/kernel/cpu/mce/therm_throt.c | 5 ++--- arch/x86/kernel/cpu/mce/threshold.c | 5 ++--- arch/x86/kernel/idt.c | 28 +++++++++++++------------- arch/x86/kernel/irq_work.c | 6 +++--- arch/x86/platform/uv/tlb_uv.c | 2 +- 14 files changed, 48 insertions(+), 102 deletions(-)
arch/x86/entry/entry_64.S +0 -19
@@ -956,9 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm -#ifdef CONFIG_X86_UV -apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt -#endif #ifdef CONFIG_HAVE_KVM apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi
@@ -966,26 +963,10 @@ apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_post apicinterrupt3 POSTED_INTR_NESTED_VECTOR kvm_posted_intr_nested_ipi smp_kvm_posted_intr_nested_ipi #endif -#ifdef CONFIG_X86_MCE_THRESHOLD -apicinterrupt THRESHOLD_APIC_VECTOR threshold_interrupt smp_threshold_interrupt -#endif - -#ifdef CONFIG_X86_MCE_AMD -apicinterrupt DEFERRED_ERROR_VECTOR deferred_error_interrupt smp_deferred_error_interrupt -#endif - -#ifdef CONFIG_X86_THERMAL_VECTOR -apicinterrupt THERMAL_APIC_VECTOR thermal_interrupt smp_thermal_interrupt -#endif - #ifdef CONFIG_SMP apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif -#ifdef CONFIG_IRQ_WORK -apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt -#endif - /* * Reload gs selector with exception handling * edi: new selector
arch/x86/include/asm/apic.h +0 -13
@@ -534,24 +534,11 @@ static inline void entering_ack_irq(void) ack_APIC_irq(); } -static inline void ipi_entering_ack_irq (void ) -{ -irq_enter (); -ack_APIC_irq (); -kvm_set_cpu_l1tf_flush_l1d (); -} -static inline void exiting_irq (void ) {irq_exit (); } -static inline void exiting_ack_irq (void ) -{ -ack_APIC_irq (); -irq_exit (); -} -extern void ioapic_zap_locks (void );#endif /* _ASM_X86_APIC_H */
arch/x86/include/asm/entry_arch.h +0 -25
@@ -20,28 +20,3 @@ BUILD_INTERRUPT(kvm_posted_intr_wakeup_ipi, POSTED_INTR_WAKEUP_VECTOR) BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)#endif -/* -* every pentium local APIC has two 'local interrupts', with a -* soft-definable vector attached to both interrupts, one of -* which is a timer interrupt, the other one is error counter -* overflow. Linux uses the local APIC timer interrupt to get -* a much simpler SMP time architecture: -*/ -#ifdef CONFIG_X86_LOCAL_APIC - -#ifdef CONFIG_IRQ_WORK -BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR) -#endif - -#ifdef CONFIG_X86_THERMAL_VECTOR -BUILD_INTERRUPT(thermal_interrupt,THERMAL_APIC_VECTOR) -#endif - -#ifdef CONFIG_X86_MCE_THRESHOLD -BUILD_INTERRUPT(threshold_interrupt,THRESHOLD_APIC_VECTOR) -#endif - -#ifdef CONFIG_X86_MCE_AMD -BUILD_INTERRUPT(deferred_error_interrupt, DEFERRED_ERROR_VECTOR) -#endif -#endif
arch/x86/include/asm/hw_irq.h +0 -6
@@ -32,15 +32,9 @@ extern asmlinkage void kvm_posted_intr_ipi(void); extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); extern asmlinkage void kvm_posted_intr_nested_ipi(void); -extern asmlinkage void irq_work_interrupt(void); -extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void); -extern asmlinkage void threshold_interrupt(void); -extern asmlinkage void deferred_error_interrupt(void); -#ifdef CONFIG_X86_LOCAL_APIC struct irq_data ;struct pci_dev ;
arch/x86/include/asm/idtentry.h +22 -0
@@ -584,6 +584,28 @@ DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR, sysvec_call_function_singleDECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_VECTOR , sysvec_call_function);#endif +#ifdef CONFIG_X86_LOCAL_APIC +# ifdef CONFIG_X86_UV +DECLARE_IDTENTRY_SYSVEC (UV_BAU_MESSAGE , sysvec_uv_bau_message); +# endif + +# ifdef CONFIG_X86_MCE_THRESHOLD +DECLARE_IDTENTRY_SYSVEC (THRESHOLD_APIC_VECTOR , sysvec_threshold); +# endif + +# ifdef CONFIG_X86_MCE_AMD +DECLARE_IDTENTRY_SYSVEC (DEFERRED_ERROR_VECTOR , sysvec_deferred_error); +# endif + +# ifdef CONFIG_X86_THERMAL_VECTOR +DECLARE_IDTENTRY_SYSVEC (THERMAL_APIC_VECTOR , sysvec_thermal); +# endif + +# ifdef CONFIG_IRQ_WORK +DECLARE_IDTENTRY_SYSVEC (IRQ_WORK_VECTOR , sysvec_irq_work); +# endif +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq_work.h +0 -1
@@ -10,7 +10,6 @@ static inline bool arch_irq_work_has_interrupt(void) return boot_cpu_has(X86_FEATURE_APIC); }extern void arch_irq_work_raise (void ); -extern __visible void smp_irq_work_interrupt(struct pt_regs *regs);#else static inline bool arch_irq_work_has_interrupt (void ) {
arch/x86/include/asm/traps.h +0 -5
@@ -34,11 +34,6 @@ static inline int get_si_code(unsigned long condition)extern int panic_on_unrecovered_nmi;void math_emulate (struct math_emu_info *); -#ifndef CONFIG_X86_32 -asmlinkage void smp_thermal_interrupt(struct pt_regs *regs); -asmlinkage void smp_threshold_interrupt(struct pt_regs *regs); -asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs); -#endif #ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message,
arch/x86/include/asm/uv/uv_bau.h +2 -6
@@ -12,6 +12,8 @@#define _ASM_X86_UV_UV_BAU_H#include <linux/bitmap.h> +#include <asm/idtentry.h> +#define BITSPERBYTE 8/*
@@ -799,12 +801,6 @@ static inline void bau_cpubits_clear(struct bau_local_cpumask *dstp, int nbits)bitmap_zero (&dstp->bits, nbits); } -extern void uv_bau_message_intr1 (void ); -#ifdef CONFIG_TRACING -#define trace_uv_bau_message_intr1 uv_bau_message_intr1 -#endif -extern void uv_bau_timeout_intr1 (void ); -struct atomic_short {short counter; };
arch/x86/kernel/cpu/mce/amd.c +2 -3
@@ -907,14 +907,13 @@ static void __log_error(unsigned int bank, u64 status, u64 addr, u64 misc)mce_log (&m); } -asmlinkage __visible void __irq_entry smp_deferred_error_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_deferred_error) { -entering_irq ();trace_deferred_error_apic_entry (DEFERRED_ERROR_VECTOR );inc_irq_stat (irq_deferred_error_count);deferred_error_int_vector ();trace_deferred_error_apic_exit (DEFERRED_ERROR_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }/*
arch/x86/kernel/cpu/mce/therm_throt.c +2 -3
@@ -614,14 +614,13 @@ static void unexpected_thermal_interrupt(void)static void (*smp_thermal_vector)(void ) = unexpected_thermal_interrupt; -asmlinkage __visible void __irq_entry smp_thermal_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_thermal) { -entering_irq ();trace_thermal_apic_entry (THERMAL_APIC_VECTOR );inc_irq_stat (irq_thermal_count);smp_thermal_vector ();trace_thermal_apic_exit (THERMAL_APIC_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }/* Thermal monitoring depends on APIC, ACPI and clock modulation */
arch/x86/kernel/cpu/mce/threshold.c +2 -3
@@ -21,12 +21,11 @@ static void default_threshold_interrupt(void)void (*mce_threshold_vector)(void ) = default_threshold_interrupt; -asmlinkage __visible void __irq_entry smp_threshold_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_threshold) { -entering_irq ();trace_threshold_apic_entry (THRESHOLD_APIC_VECTOR );inc_irq_stat (irq_threshold_count);mce_threshold_vector ();trace_threshold_apic_exit (THRESHOLD_APIC_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }
arch/x86/kernel/idt.c +14 -14
@@ -120,33 +120,33 @@ static const __initconst struct idt_data apic_idts[] = {#endif #ifdef CONFIG_X86_THERMAL_VECTOR - INTG(THERMAL_APIC_VECTOR, thermal_interrupt), + INTG(THERMAL_APIC_VECTOR, asm_sysvec_thermal),#endif #ifdef CONFIG_X86_MCE_THRESHOLD - INTG(THRESHOLD_APIC_VECTOR, threshold_interrupt), + INTG(THRESHOLD_APIC_VECTOR, asm_sysvec_threshold),#endif #ifdef CONFIG_X86_MCE_AMD - INTG(DEFERRED_ERROR_VECTOR, deferred_error_interrupt), + INTG(DEFERRED_ERROR_VECTOR, asm_sysvec_deferred_error),#endif #ifdef CONFIG_X86_LOCAL_APIC - INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), - INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi), + INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), + INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi), # ifdef CONFIG_HAVE_KVM - INTG(POSTED_INTR_VECTOR, kvm_posted_intr_ipi), - INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi), - INTG(POSTED_INTR_NESTED_VECTOR, kvm_posted_intr_nested_ipi), + INTG(POSTED_INTR_VECTOR, kvm_posted_intr_ipi), + INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi), + INTG(POSTED_INTR_NESTED_VECTOR, kvm_posted_intr_nested_ipi), # endif # ifdef CONFIG_IRQ_WORK - INTG(IRQ_WORK_VECTOR, irq_work_interrupt), + INTG(IRQ_WORK_VECTOR, asm_sysvec_irq_work), # endif -#ifdef CONFIG_X86_UV - INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), -#endif - INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), - INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), +# ifdef CONFIG_X86_UV + INTG(UV_BAU_MESSAGE, asm_sysvec_uv_bau_message), +# endif + INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), + INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), #endif };
arch/x86/kernel/irq_work.c +3 -3
@@ -9,18 +9,18 @@#include <linux/irq_work.h> #include <linux/hardirq.h> #include <asm/apic.h> +#include <asm/idtentry.h> #include <asm/trace/irq_vectors.h> #include <linux/interrupt.h> #ifdef CONFIG_X86_LOCAL_APIC -__visible void __irq_entry smp_irq_work_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_irq_work) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_irq_work_entry (IRQ_WORK_VECTOR );inc_irq_stat (apic_irq_work_irqs);irq_work_run ();trace_irq_work_exit (IRQ_WORK_VECTOR ); -exiting_irq (); }void arch_irq_work_raise (void )
arch/x86/platform/uv/tlb_uv.c +1 -1
@@ -1272,7 +1272,7 @@ static void process_uv2_message(struct msg_desc *mdp, struct bau_control *bcp)* (the resource will not be freed until noninterruptable cpus see this * interrupt; hardware may timeout the s/w ack and reply ERROR) */ -void uv_bau_message_interrupt (struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_uv_bau_message) {int count =0 ;cycles_t time_start;
PATCH v9 29/39 x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC*
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Convert KVM specific system vectors to IDTENTRY_SYSVEC*:
The two empty stub handlers which only increment the stats counter do no need to run on the interrupt stack. Use IDTENTRY_SYSVEC_SIMPLE for them.
The wakeup handler does more work and runs on the interrupt stack.
None of these handlers need to save and restore the irq_regs pointer.
--- arch/x86/entry/entry_64.S | 7 ------- arch/x86/include/asm/entry_arch.h | 7 ------- arch/x86/include/asm/hw_irq.h | 4 ---- arch/x86/include/asm/idtentry.h | 6 ++++++ arch/x86/include/asm/irq.h | 3 --- arch/x86/kernel/idt.c | 6 +++--- arch/x86/kernel/irq.c | 24 ++++++------------------ 7 files changed, 15 insertions(+), 42 deletions(-)
arch/x86/entry/entry_64.S +0 -7
@@ -956,13 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm - -#ifdef CONFIG_HAVE_KVM -apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi -apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_posted_intr_wakeup_ipi -apicinterrupt3 POSTED_INTR_NESTED_VECTOR kvm_posted_intr_nested_ipi smp_kvm_posted_intr_nested_ipi -#endif - #ifdef CONFIG_SMP apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -7
@@ -13,10 +13,3 @@#ifdef CONFIG_SMP BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)#endif - -#ifdef CONFIG_HAVE_KVM -BUILD_INTERRUPT(kvm_posted_intr_ipi, POSTED_INTR_VECTOR) -BUILD_INTERRUPT(kvm_posted_intr_wakeup_ipi, POSTED_INTR_WAKEUP_VECTOR) -BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR) -#endif -
arch/x86/include/asm/hw_irq.h +0 -4
@@ -29,10 +29,6 @@#include <asm/sections.h> /* Interrupt handlers registered during init_IRQ */ -extern asmlinkage void kvm_posted_intr_ipi(void); -extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); -extern asmlinkage void kvm_posted_intr_nested_ipi(void); - extern asmlinkage void reschedule_interrupt(void);#ifdef CONFIG_X86_LOCAL_APIC
arch/x86/include/asm/idtentry.h +6 -0
@@ -602,6 +602,12 @@ DECLARE_IDTENTRY_SYSVEC(IRQ_WORK_VECTOR,# endif #endif +#ifdef CONFIG_HAVE_KVM +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_VECTOR , sysvec_kvm_posted_intr_ipi); +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_WAKEUP_VECTOR , sysvec_kvm_posted_intr_wakeup_ipi); +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_NESTED_VECTOR , sysvec_kvm_posted_intr_nested_ipi); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq.h +0 -3
@@ -26,9 +26,6 @@ extern void fixup_irqs(void);#ifdef CONFIG_HAVE_KVM extern void kvm_set_posted_intr_wakeup_handler (void (*handler)(void )); -extern __visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs); -extern __visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs); -extern __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs);#endif extern void (*x86_platform_ipi_callback)(void );
arch/x86/kernel/idt.c +3 -3
@@ -135,9 +135,9 @@ static const __initconst struct idt_data INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi),# ifdef CONFIG_HAVE_KVM -INTG (POSTED_INTR_VECTOR , kvm_posted_intr_ipi), -INTG (POSTED_INTR_WAKEUP_VECTOR , kvm_posted_intr_wakeup_ipi), -INTG (POSTED_INTR_NESTED_VECTOR , kvm_posted_intr_nested_ipi), +INTG (POSTED_INTR_VECTOR , asm_sysvec_kvm_posted_intr_ipi), +INTG (POSTED_INTR_WAKEUP_VECTOR , asm_sysvec_kvm_posted_intr_wakeup_ipi), +INTG (POSTED_INTR_NESTED_VECTOR , asm_sysvec_kvm_posted_intr_nested_ipi), # endif # ifdef CONFIG_IRQ_WORK INTG(IRQ_WORK_VECTOR, asm_sysvec_irq_work),
arch/x86/kernel/irq.c +6 -18
@@ -298,41 +298,29 @@ EXPORT_SYMBOL_GPL(kvm_set_posted_intr_wa/* * Handler for POSTED_INTERRUPT_VECTOR. */ -__visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC_SIMPLE (sysvec_kvm_posted_intr_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_ipis); -exiting_irq (); -set_irq_regs (old_regs); }/* * Handler for POSTED_INTERRUPT_WAKEUP_VECTOR. */ -__visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_kvm_posted_intr_wakeup_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_wakeup_ipis);kvm_posted_intr_wakeup_handler (); -exiting_irq (); -set_irq_regs (old_regs); }/* * Handler for POSTED_INTERRUPT_NESTED_VECTOR. */ -__visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC_SIMPLE (sysvec_kvm_posted_intr_nested_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_nested_ipis); -exiting_irq (); -set_irq_regs (old_regs); }#endif
[tip: x86/entry] x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC*
The following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 79e5785846099b7f48c92ffa299a4ddc1e4a98f6 Gitweb: https://git.kernel.org/tip/79e5785846099b7f48c92ffa299a4ddc1e4a98f6 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:42 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:28 +02:00
x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC*
Convert KVM specific system vectors to IDTENTRY_SYSVEC*:
The two empty stub handlers which only increment the stats counter do no need to run on the interrupt stack. Use IDTENTRY_SYSVEC_SIMPLE for them.
The wakeup handler does more work and runs on the interrupt stack.
None of these handlers need to save and restore the irq_regs pointer.
--- arch/x86/entry/entry_64.S | 7 ------- arch/x86/include/asm/entry_arch.h | 7 ------- arch/x86/include/asm/hw_irq.h | 4 ---- arch/x86/include/asm/idtentry.h | 6 ++++++ arch/x86/include/asm/irq.h | 3 --- arch/x86/kernel/idt.c | 6 +++--- arch/x86/kernel/irq.c | 24 ++++++------------------ 7 files changed, 15 insertions(+), 42 deletions(-)
arch/x86/entry/entry_64.S +0 -7
@@ -956,13 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm - -#ifdef CONFIG_HAVE_KVM -apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi -apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_posted_intr_wakeup_ipi -apicinterrupt3 POSTED_INTR_NESTED_VECTOR kvm_posted_intr_nested_ipi smp_kvm_posted_intr_nested_ipi -#endif - #ifdef CONFIG_SMP apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -7
@@ -13,10 +13,3 @@#ifdef CONFIG_SMP BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)#endif - -#ifdef CONFIG_HAVE_KVM -BUILD_INTERRUPT(kvm_posted_intr_ipi, POSTED_INTR_VECTOR) -BUILD_INTERRUPT(kvm_posted_intr_wakeup_ipi, POSTED_INTR_WAKEUP_VECTOR) -BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR) -#endif -
arch/x86/include/asm/hw_irq.h +0 -4
@@ -29,10 +29,6 @@#include <asm/sections.h> /* Interrupt handlers registered during init_IRQ */ -extern asmlinkage void kvm_posted_intr_ipi(void); -extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); -extern asmlinkage void kvm_posted_intr_nested_ipi(void); - extern asmlinkage void reschedule_interrupt(void);#ifdef CONFIG_X86_LOCAL_APIC
arch/x86/include/asm/idtentry.h +6 -0
@@ -606,6 +606,12 @@ DECLARE_IDTENTRY_SYSVEC(IRQ_WORK_VECTOR, sysvec_irq_work);# endif #endif +#ifdef CONFIG_HAVE_KVM +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_VECTOR , sysvec_kvm_posted_intr_ipi); +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_WAKEUP_VECTOR , sysvec_kvm_posted_intr_wakeup_ipi); +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_NESTED_VECTOR , sysvec_kvm_posted_intr_nested_ipi); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq.h +0 -3
@@ -26,9 +26,6 @@ extern void fixup_irqs(void);#ifdef CONFIG_HAVE_KVM extern void kvm_set_posted_intr_wakeup_handler (void (*handler)(void )); -extern __visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs); -extern __visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs); -extern __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs);#endif extern void (*x86_platform_ipi_callback)(void );
arch/x86/kernel/idt.c +3 -3
@@ -135,9 +135,9 @@ static const __initconst struct idt_data apic_idts[] = { INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi),# ifdef CONFIG_HAVE_KVM -INTG (POSTED_INTR_VECTOR , kvm_posted_intr_ipi), -INTG (POSTED_INTR_WAKEUP_VECTOR , kvm_posted_intr_wakeup_ipi), -INTG (POSTED_INTR_NESTED_VECTOR , kvm_posted_intr_nested_ipi), +INTG (POSTED_INTR_VECTOR , asm_sysvec_kvm_posted_intr_ipi), +INTG (POSTED_INTR_WAKEUP_VECTOR , asm_sysvec_kvm_posted_intr_wakeup_ipi), +INTG (POSTED_INTR_NESTED_VECTOR , asm_sysvec_kvm_posted_intr_nested_ipi), # endif # ifdef CONFIG_IRQ_WORK INTG(IRQ_WORK_VECTOR, asm_sysvec_irq_work),
arch/x86/kernel/irq.c +6 -18
@@ -298,41 +298,29 @@ EXPORT_SYMBOL_GPL(kvm_set_posted_intr_wakeup_handler);/* * Handler for POSTED_INTERRUPT_VECTOR. */ -__visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC_SIMPLE (sysvec_kvm_posted_intr_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_ipis); -exiting_irq (); -set_irq_regs (old_regs); }/* * Handler for POSTED_INTERRUPT_WAKEUP_VECTOR. */ -__visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_kvm_posted_intr_wakeup_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_wakeup_ipis);kvm_posted_intr_wakeup_handler (); -exiting_irq (); -set_irq_regs (old_regs); }/* * Handler for POSTED_INTERRUPT_NESTED_VECTOR. */ -__visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC_SIMPLE (sysvec_kvm_posted_intr_nested_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_nested_ipis); -exiting_irq (); -set_irq_regs (old_regs); }#endif
PATCH v9 30/39 x86/entry: Convert various hypervisor vectors to IDTENTRY_SYSVEC
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Convert various hypervisor vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
No functional change.
--- arch/x86/entry/entry_32.S | 14 -------------- arch/x86/entry/entry_64.S | 17 ----------------- arch/x86/hyperv/hv_init.c | 9 +++------ arch/x86/include/asm/acrn.h | 11 ----------- arch/x86/include/asm/apic.h | 20 -------------------- arch/x86/include/asm/idtentry.h | 10 ++++++++++ arch/x86/include/asm/mshyperv.h | 13 ------------- arch/x86/kernel/cpu/acrn.c | 9 ++++----- arch/x86/kernel/cpu/mshyperv.c | 22 ++++++++++------------ 9 files changed, 27 insertions(+), 98 deletions(-)
arch/x86/entry/entry_32.S +0 -14
@@ -1342,20 +1342,6 @@ BUILD_INTERRUPT3(xen_hvm_callback_vector xen_evtchn_do_upcall) #endif - -#if IS_ENABLED(CONFIG_HYPERV) - -BUILD_INTERRUPT3(hyperv_callback_vector, HYPERVISOR_CALLBACK_VECTOR, - hyperv_vector_handler) - -BUILD_INTERRUPT3(hyperv_reenlightenment_vector, HYPERV_REENLIGHTENMENT_VECTOR, - hyperv_reenlightenment_intr) - -BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR, - hv_stimer0_vector_handler) - -#endif /* CONFIG_HYPERV */ - SYM_CODE_START_LOCAL_NOALIGN(handle_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -17
@@ -1116,23 +1116,6 @@ apicinterrupt3 HYPERVISOR_CALLBACK_VECTO xen_hvm_callback_vector xen_evtchn_do_upcall #endif - -#if IS_ENABLED(CONFIG_HYPERV) -apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \ - hyperv_callback_vector hyperv_vector_handler - -apicinterrupt3 HYPERV_REENLIGHTENMENT_VECTOR \ - hyperv_reenlightenment_vector hyperv_reenlightenment_intr - -apicinterrupt3 HYPERV_STIMER0_VECTOR \ - hv_stimer0_callback_vector hv_stimer0_vector_handler -#endif /* CONFIG_HYPERV */ - -#if IS_ENABLED(CONFIG_ACRN_GUEST) -apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \ - acrn_hv_callback_vector acrn_hv_vector_handler -#endif - /* * Save all registers in pt_regs, and switch gs if needed. * Use slow, but surefire "are we in kernel?" check.
arch/x86/hyperv/hv_init.c +3 -6
@@ -15,6 +15,7 @@#include <asm/hypervisor.h> #include <asm/hyperv-tlfs.h> #include <asm/mshyperv.h> +#include <asm/idtentry.h> #include <linux/version.h> #include <linux/vmalloc.h> #include <linux/mm.h>
@@ -153,15 +154,11 @@ static inline bool hv_reenlightenment_av ms_hyperv.features &HV_X64_ACCESS_REENLIGHTENMENT ; } -__visible void __irq_entry hyperv_reenlightenment_intr(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_hyperv_reenlightenment) { -entering_ack_irq (); - +ack_APIC_irq ();inc_irq_stat (irq_hv_reenlightenment_count); -schedule_delayed_work (&hv_reenlightenment_work,HZ /10 ); - -exiting_irq (); }void set_hv_tscchange_cb (void (*cb)(void ))
arch/x86/include/asm/acrn.h deleted +0 -11
@@ -1,11 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_X86_ACRN_H -#define _ASM_X86_ACRN_H - -extern void acrn_hv_callback_vector (void ); -#ifdef CONFIG_TRACING -#define trace_acrn_hv_callback_vector acrn_hv_callback_vector -#endif - -extern void acrn_hv_vector_handler (struct pt_regs *regs); -#endif /* _ASM_X86_ACRN_H */
arch/x86/include/asm/apic.h +0 -20
@@ -519,26 +519,6 @@ static inline bool apic_id_is_primary_thstatic inline void apic_smt_update (void ) { }#endif -extern void irq_enter (void ); -extern void irq_exit (void ); - -static inline void entering_irq (void ) -{ -irq_enter (); -kvm_set_cpu_l1tf_flush_l1d (); -} - -static inline void entering_ack_irq (void ) -{ -entering_irq (); -ack_APIC_irq (); -} - -static inline void exiting_irq (void ) -{ -irq_exit (); -} -extern void ioapic_zap_locks (void );#endif /* _ASM_X86_APIC_H */
arch/x86/include/asm/idtentry.h +10 -0
@@ -608,6 +608,16 @@ DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_WAKEDECLARE_IDTENTRY_SYSVEC (POSTED_INTR_NESTED_VECTOR , sysvec_kvm_posted_intr_nested_ipi);#endif +#if IS_ENABLED (CONFIG_HYPERV ) +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_CALLBACK_VECTOR , sysvec_hyperv_callback); +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_REENLIGHTENMENT_VECTOR , sysvec_hyperv_reenlightenment); +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_STIMER0_VECTOR , sysvec_hyperv_stimer0); +#endif + +#if IS_ENABLED (CONFIG_ACRN_GUEST ) +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_CALLBACK_VECTOR , sysvec_acrn_hv_callback); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/mshyperv.h +0 -13
@@ -54,20 +54,8 @@ typedef int (*hyperv_fill_flush_list_funvclocks_set_used (VDSO_CLOCKMODE_HVCLOCK );#define hv_get_raw_timer () rdtsc_ordered() -void hyperv_callback_vector (void ); -void hyperv_reenlightenment_vector (void ); -#ifdef CONFIG_TRACING -#define trace_hyperv_callback_vector hyperv_callback_vector -#endif void hyperv_vector_handler (struct pt_regs *regs); -/* -* Routines for stimer0 Direct Mode handling. -* On x86/x64, there are no percpu actions to take. -*/ -void hv_stimer0_vector_handler (struct pt_regs *regs); -void hv_stimer0_callback_vector (void ); -static inline void hv_enable_stimer0_percpu_irq (int irq) {}static inline void hv_disable_stimer0_percpu_irq (int irq) {}
@@ -226,7 +214,6 @@ void hyperv_setup_mmu_ops(void);void *hv_alloc_hyperv_page (void );void *hv_alloc_hyperv_zeroed_page (void );void hv_free_hyperv_page (unsigned long addr); -void hyperv_reenlightenment_intr (struct pt_regs *regs);void set_hv_tscchange_cb (void (*cb)(void ));void clear_hv_tscchange_cb (void );void hyperv_stop_tsc_emulation (void );
arch/x86/kernel/cpu/acrn.c +4 -5
@@ -10,10 +10,10 @@*/ #include <linux/interrupt.h> -#include <asm/acrn.h> #include <asm/apic.h> #include <asm/desc.h> #include <asm/hypervisor.h> +#include <asm/idtentry.h> #include <asm/irq_regs.h> static uint32_t __init acrn_detect (void )
@@ -24,7 +24,7 @@ static uint32_t __init acrn_detect(void)static void __init acrn_init_platform (void ) {/* Setup the IDT for ACRN hypervisor callback */ -alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , acrn_hv_callback_vector); +alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , asm_sysvec_acrn_hv_callback); }static bool acrn_x2apic_available (void )
@@ -39,7 +39,7 @@ static bool acrn_x2apic_available(void)static void (*acrn_intr_handler)(void ); -__visible void __irq_entry acrn_hv_vector_handler(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_acrn_hv_callback) {struct pt_regs *old_regs =set_irq_regs (regs);
@@ -50,13 +50,12 @@ static void (*acrn_intr_handler)(void);* will block the interrupt whose vector is lower than * HYPERVISOR_CALLBACK_VECTOR. */ -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (irq_hv_callback_count);if (acrn_intr_handler)acrn_intr_handler (); -exiting_irq ();set_irq_regs (old_regs); }
arch/x86/kernel/cpu/mshyperv.c +10 -12
@@ -23,6 +23,7 @@#include <asm/hyperv-tlfs.h> #include <asm/mshyperv.h> #include <asm/desc.h> +#include <asm/idtentry.h> #include <asm/irq_regs.h> #include <asm/i8259.h> #include <asm/apic.h>
@@ -40,11 +41,10 @@ static void (*hv_stimer0_handler)(void);static void (*hv_kexec_handler)(void );static void (*hv_crash_handler)(struct pt_regs *regs); -__visible void __irq_entry hyperv_vector_handler(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_hyperv_callback) {struct pt_regs *old_regs =set_irq_regs (regs); -entering_irq ();inc_irq_stat (irq_hv_callback_count);if (vmbus_handler)vmbus_handler ();
@@ -52,7 +52,6 @@ static void (*hv_crash_handler)(struct pif (ms_hyperv.hints &HV_DEPRECATING_AEOI_RECOMMENDED )ack_APIC_irq (); -exiting_irq ();set_irq_regs (old_regs); }
@@ -73,19 +72,16 @@ EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);* Routines to do per-architecture handling of stimer0 * interrupts when in Direct Mode */ - -__visible void __irq_entry hv_stimer0_vector_handler(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_hyperv_stimer0) {struct pt_regs *old_regs =set_irq_regs (regs); -entering_irq ();inc_irq_stat (hyperv_stimer0_count);if (hv_stimer0_handler)hv_stimer0_handler ();add_interrupt_randomness (HYPERV_STIMER0_VECTOR ,0 );ack_APIC_irq (); -exiting_irq ();set_irq_regs (old_regs); }
@@ -331,17 +327,19 @@ static void __init ms_hyperv_init_platfo x86_platform.apic_post_init = hyperv_init;hyperv_setup_mmu_ops ();/* Setup the IDT for hypervisor callback */ -alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , hyperv_callback_vector); +alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , sysvec_hyperv_callback);/* Setup the IDT for reenlightenment notifications */ -if (ms_hyperv.features &HV_X64_ACCESS_REENLIGHTENMENT ) +if (ms_hyperv.features &HV_X64_ACCESS_REENLIGHTENMENT ) {alloc_intr_gate (HYPERV_REENLIGHTENMENT_VECTOR , - hyperv_reenlightenment_vector); + asm_sysvec_hyperv_reenlightenment); + }/* Setup the IDT for stimer0 */ -if (ms_hyperv.misc_features &HV_STIMER_DIRECT_MODE_AVAILABLE ) +if (ms_hyperv.misc_features &HV_STIMER_DIRECT_MODE_AVAILABLE ) {alloc_intr_gate (HYPERV_STIMER0_VECTOR , - hv_stimer0_callback_vector); + asm_sysvec_hyperv_stimer0); + }# ifdef CONFIG_SMP smp_ops.smp_prepare_boot_cpu = hv_smp_prepare_boot_cpu;
Re: [patch V9 30/39] x86/entry: Convert various hypervisor vectors to IDTENTRY_SYSVEC
Quoted 9 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Convert various hypervisor vectors to
IDTENTRY_SYSVEC
- Implement the C entry point withDEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub withDECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove theBUILD_INTERRUPTentries in 32bit
- Remove the old prototypes
Quoted 4 lines
No functional change.
Signed-off-by Thomas Gleixner <[email protected]>Acked-by Andy Lutomirski <[email protected]>
For Hyper-V bits: