Skip to contentSkip to conversation

x86/entry: Rework leftovers (was part V)

PATCH Applied Thomas Gleixner 16 participants 132 messages May 21, 2020 – Aug 26, 2020 Page 1 of 2

Subsystems (11)
lore
Share & tools
lore
#0 lore raw

v9 cover x86/entry: Rework leftovers (was part V)

Folks!

This is V9 of the rework series. V7 and V8 were never posted but I used the version numbers for tags while fixing up 0day complaints. The last posted version was V6 which can be found here:

https://lore.kernel.org/r/[email protected]

The V9 leftover series is based on:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry

That branch contains the merged part 1-4 of the original 5 part series.

V9 has the following changes vs. V6:

- Rebase on tip x86/entry

- Simplified the hardware latency detector changes by moving the invocation to the right place in nmi_enter/exit() and annotate it.

- Reworked the conditional RCU handling so it is now used unconditionally everywhere. That simplified the idtentry_enter/exit code significantly and also allowed to simplify the XEN hypercall voluntary preemption handling.

- Moved the run on irq stack logic into an inline to avoid having the same conditionals all over the place and fixed up the relevant places.

- Picked up Acked-by and Reviewed-by tags where appropriate.

The full series is available from:

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-v9-the-rest

If we agree on the RCU changes, then these will be applied into the core/rcu branch first so Paul can pick them up to avoid the next conflict horrors.

Thanks,

tglx

#1 lore raw

PATCH v9 1/39 nmi, tracing: Make hardware latency tracing noinstr safe

The hardware latency tracer calls into instrumentable functions. Move the calls into the RCU watching sections and annotate them.

---
V9: Simplified version

---
 include/linux/hardirq.h |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
include/linux/hardirq.h +6 -2
@@ -82,20 +82,24 @@ extern void irq_exit(void);
 		arch_nmi_enter();				\
 		printk_nmi_enter();				\
 		lockdep_off();					\
-		ftrace_nmi_enter();				\
 		BUG_ON(in_nmi() == NMI_MASK);			\
 		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
 		rcu_nmi_enter();				\
 		lockdep_hardirq_enter();			\
+		instrumentation_begin();			\
+		ftrace_nmi_enter();				\
+		instrumentation_end();				\
 	} while (0)
 
 #define nmi_exit()						\
 	do {							\
+		instrumentation_begin();			\
+		ftrace_nmi_exit();				\
+		instrumentation_end();				\
 		lockdep_hardirq_exit();				\
 		rcu_nmi_exit();					\
 		BUG_ON(!in_nmi());				\
 		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
-		ftrace_nmi_exit();				\
 		lockdep_on();					\
 		printk_nmi_exit();				\
 		arch_nmi_exit();				\
#2 lore raw

[tip: x86/entry] nmi, tracing: Make hardware latency tracing noinstr safe

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     1e826b1d2265d1e8ba5f51827a8dba2aa0295f6c
Gitweb:        https://git.kernel.org/tip/1e826b1d2265d1e8ba5f51827a8dba2aa0295f6c
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:14 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

nmi, tracing: Make hardware latency tracing noinstr safe

The hardware latency tracer calls into instrumentable functions. Move the calls into the RCU watching sections and annotate them.

---
 include/linux/hardirq.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
include/linux/hardirq.h +6 -2
index e07cf85..29b862a 100644
@@ -87,20 +87,24 @@ extern void rcu_nmi_exit(void);
 		arch_nmi_enter();				\
 		printk_nmi_enter();				\
 		lockdep_off();					\
-		ftrace_nmi_enter();				\
 		BUG_ON(in_nmi() == NMI_MASK);			\
 		__preempt_count_add(NMI_OFFSET + HARDIRQ_OFFSET);	\
 		rcu_nmi_enter();				\
 		lockdep_hardirq_enter();			\
+		instrumentation_begin();			\
+		ftrace_nmi_enter();				\
+		instrumentation_end();				\
 	} while (0)
 
 #define nmi_exit()						\
 	do {							\
+		instrumentation_begin();			\
+		ftrace_nmi_exit();				\
+		instrumentation_end();				\
 		lockdep_hardirq_exit();				\
 		rcu_nmi_exit();					\
 		BUG_ON(!in_nmi());				\
 		__preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET);	\
-		ftrace_nmi_exit();				\
 		lockdep_on();					\
 		printk_nmi_exit();				\
 		arch_nmi_exit();				\
#3 lore raw

PATCH v9 2/39 rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()

In reply to Thomas Gleixner

From: Paul E. McKenney <[email protected]>

There will likely be exception handlers that can sleep, which rules out the usual approach of invoking rcu_nmi_enter() on entry and also rcu_nmi_exit() on all exit paths. However, the alternative approach of just not calling anything can prevent RCU from coaxing quiescent states from nohz_full CPUs that are looping in the kernel: RCU must instead IPI them explicitly. It would be better to enable the scheduler tick on such CPUs to interact with RCU in a lighter-weight manner, and this enabling is one of the things that rcu_nmi_enter() currently does.

What is needed is something that helps RCU coax quiescent states while not preventing subsequent sleeps. This commit therefore splits out the nohz_full scheduler-tick enabling from the rest of the rcu_nmi_enter() logic into a new function named rcu_irq_enter_check_tick().

[ tglx: Renamed the function and made it a nop when context tracking is off ]

---
V9: New patch

---
 include/linux/hardirq.h |    9 +++++
 kernel/rcu/tree.c       |   82 ++++++++++++++++++++++++++++++++++++------------
 2 files changed, 71 insertions(+), 20 deletions(-)
include/linux/hardirq.h +9 -0
@@ -2,6 +2,7 @@
 #ifndef LINUX_HARDIRQ_H
 #define LINUX_HARDIRQ_H
 
+#include <linux/context_tracking_state.h>
 #include <linux/preempt.h>
 #include <linux/lockdep.h>
 #include <linux/ftrace_irq.h>
@@ -27,6 +28,14 @@ extern void rcu_nmi_enter(void);
 extern void rcu_nmi_exit(void);
 #endif
 
+void __rcu_irq_enter_check_tick(void);
+
+static __always_inline void rcu_irq_enter_check_tick(void)
+{
+	if (context_tracking_enabled())
+		__rcu_irq_enter_check_tick();
+}
+
 /*
  * It is safe to do non-atomic ops on ->hardirq_context,
  * because NMI handlers may not preempt and the ops are
kernel/rcu/tree.c +62 -20

94 lines not on this page · raw message

maintainer
#4 lore raw

Re: [patch V9 02/39] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()

Quoted 11 lines · Thomas Gleixner

From: Paul E. McKenney <[email protected]>

There will likely be exception handlers that can sleep, which rules out the usual approach of invoking rcu_nmi_enter() on entry and also rcu_nmi_exit() on all exit paths. However, the alternative approach of just not calling anything can prevent RCU from coaxing quiescent states from nohz_full CPUs that are looping in the kernel: RCU must instead IPI them explicitly. It would be better to enable the scheduler tick on such CPUs to interact with RCU in a lighter-weight manner, and this enabling is one of the things that rcu_nmi_enter() currently does.

Quoted 6 lines

What is needed is something that helps RCU coax quiescent states while not preventing subsequent sleeps. This commit therefore splits out the nohz_full scheduler-tick enabling from the rest of the rcu_nmi_enter() logic into a new function named rcu_irq_enter_check_tick().

[ tglx: Renamed the function and made it a nop when context tracking is off ]

The new name works for me! A couple of nits called out below.

Thanx, Paul

Quoted 21 lines

---
V9: New patch

---
 include/linux/hardirq.h |    9 +++++
 kernel/rcu/tree.c       |   82 ++++++++++++++++++++++++++++++++++++------------
 2 files changed, 71 insertions(+), 20 deletions(-)
include/linux/hardirq.h +1 -0
@@ -2,6 +2,7 @@
 #ifndef LINUX_HARDIRQ_H
 #define LINUX_HARDIRQ_H
 
+#include <linux/context_tracking_state.h>
 #include <linux/preempt.h>
 #include <linux/lockdep.h>
 #include <linux/ftrace_irq.h>

include/linux/hardirq.h extern void rcu_nmi_enter(void); · line 36

Quoted 9 lines
include/linux/hardirq.h +6 -0
@@ -27,6 +28,14 @@ extern void rcu_nmi_enter(void);
 extern void rcu_nmi_exit(void);
 #endif
 
+void __rcu_irq_enter_check_tick(void);
+
+static __always_inline void rcu_irq_enter_check_tick(void)
+{
+	if (context_tracking_enabled())
+		__rcu_irq_enter_check_tick();

I suggest moving the WARN_ON_ONCE(in_nmi()) check here to avoid calling in_nmi() twice. Because of the READ_ONCE(), the compiler cannot (had better not!) eliminate the double call.

Quoted 33 lines
include/linux/hardirq.h +2 -0
+}
+
 /*
  * It is safe to do non-atomic ops on ->hardirq_context,
  * because NMI handlers may not preempt and the ops are
kernel/rcu/tree.c +22 -0
@@ -848,6 +848,67 @@ void noinstr rcu_user_exit(void)
 {
 	rcu_eqs_exit(1);
 }
+
+/**
+ * __rcu_irq_enter_check_tick - Enable scheduler tick on CPU if RCU needs it.
+ *
+ * The scheduler tick is not normally enabled when CPUs enter the kernel
+ * from nohz_full userspace execution.  After all, nohz_full userspace
+ * execution is an RCU quiescent state and the time executing in the kernel
+ * is quite short.  Except of course when it isn't.  And it is not hard to
+ * cause a large system to spend tens of seconds or even minutes looping
+ * in the kernel, which can cause a number of problems, include RCU CPU
+ * stall warnings.
+ *
+ * Therefore, if a nohz_full CPU fails to report a quiescent state
+ * in a timely manner, the RCU grace-period kthread sets that CPU's
+ * ->rcu_urgent_qs flag with the expectation that the next interrupt or
+ * exception will invoke this function, which will turn on the scheduler
+ * tick, which will enable RCU to detect that CPU's quiescent states,
+ * for example, due to cond_resched() calls in CONFIG_PREEMPT=n kernels.
+ * The tick will be disabled once a quiescent state is reported for
+ * this CPU.
+ *
+ * Of course, in carefully tuned systems, there might never be an

kernel/rcu/tree.c void noinstr rcu_user_exit(void) · line 882

Quoted 10 lines
kernel/rcu/tree.c +10 -0
+ * interrupt or exception.  In that case, the RCU grace-period kthread
+ * will eventually cause one to happen.  However, in less carefully
+ * controlled environments, this function allows RCU to get what it
+ * needs without creating otherwise useless interruptions.
+ */
+void __rcu_irq_enter_check_tick(void)
+{
+	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
+
+	 // Enabling the tick is unsafe in NMI handlers.

There is an extra space before the "//", probably the one that used to be after the ";" below. ;-)

kernel/rcu/tree.c void noinstr rcu_user_exit(void) · line 887

Quoted 5 lines
kernel/rcu/tree.c +5 -0
+	if (WARN_ON_ONCE(in_nmi()))
+		return;
+
+	RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(),
+			 "Illegal rcu_irq_enter_check_tick() from extended quiescent state");

The instrumentation_begin() has disappeared, presumably because instrumentation is already enabled in the non-RCU code that directly calls rcu_irq_enter_check_tick(). (I do see the calls in rcu_nmi_enter() below.)

Quoted 28 lines
kernel/rcu/tree.c +24 -0
+
+	if (!tick_nohz_full_cpu(rdp->cpu) ||
+	    !READ_ONCE(rdp->rcu_urgent_qs) ||
+	    READ_ONCE(rdp->rcu_forced_tick)) {
+		// RCU doesn't need nohz_full help from this CPU, or it is
+		// already getting that help.
+		return;
+	}
+
+	// We get here only when not in an extended quiescent state and
+	// from interrupts (as opposed to NMIs).  Therefore, (1) RCU is
+	// already watching and (2) The fact that we are in an interrupt
+	// handler and that the rcu_node lock is an irq-disabled lock
+	// prevents self-deadlock.  So we can safely recheck under the lock.
+	// Note that the nohz_full state currently cannot change.
+	raw_spin_lock_rcu_node(rdp->mynode);
+	if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
+		// A nohz_full CPU is in the kernel and RCU needs a
+		// quiescent state.  Turn on the tick!
+		WRITE_ONCE(rdp->rcu_forced_tick, true);
+		tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
+	}
+	raw_spin_unlock_rcu_node(rdp->mynode);
+}
 #endif /* CONFIG_NO_HZ_FULL */
 
 /**

kernel/rcu/tree.c noinstr void rcu_nmi_enter(void) · line 956

Quoted 2 lines
kernel/rcu/tree.c +0 -0
@@ -894,26 +955,7 @@ noinstr void rcu_nmi_enter(void)
 		incby = 1;
 	} else if (!in_nmi()) {

This can just be "else" given the in_nmi() check in __rcu_irq_enter_check_tick(), right? Ah, that check got a WARN_ON_ONCE(), so never mind!

I guess that will discourage NMI-handler calls to
rcu_irq_enter_check_tick(). ;-)

It does mean a double call to in_nmi(), though, so should that WARN_ON_ONCE(in_nmi()) check go into the rcu_irq_enter_check_tick() wrapper? Or do modern compilers figure this one out? Given the READ_ONCE() in preempt_count(), I have to say that I hope not. So see my comment above on rcu_irq_enter_check_tick().

Quoted 26 lines
kernel/rcu/tree.c +1 -20
 		instrumentation_begin();
-		if (tick_nohz_full_cpu(rdp->cpu) &&
-		    rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE &&
-		    READ_ONCE(rdp->rcu_urgent_qs) &&
-		    !READ_ONCE(rdp->rcu_forced_tick)) {
-			// We get here only if we had already exited the
-			// extended quiescent state and this was an
-			// interrupt (not an NMI).  Therefore, (1) RCU is
-			// already watching and (2) The fact that we are in
-			// an interrupt handler and that the rcu_node lock
-			// is an irq-disabled lock prevents self-deadlock.
-			// So we can safely recheck under the lock.
-			raw_spin_lock_rcu_node(rdp->mynode);
-			if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
-				// A nohz_full CPU is in the kernel and RCU
-				// needs a quiescent state.  Turn on the tick!
-				WRITE_ONCE(rdp->rcu_forced_tick, true);
-				tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
-			}
-			raw_spin_unlock_rcu_node(rdp->mynode);
-		}
+		rcu_irq_enter_check_tick();
 		instrumentation_end();
 	}
 	instrumentation_begin();
#5 lore raw

Re: [patch V9 02/39] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()

Quoted 11 lines · Paul E. McKenney

include/linux/hardirq.h extern void rcu_nmi_enter(void); · line 36

Quoted 6 lines · Thomas Gleixner
include/linux/hardirq.h +6 -0
+void __rcu_irq_enter_check_tick(void);
+
+static __always_inline void rcu_irq_enter_check_tick(void)
+{
+	if (context_tracking_enabled())
+		__rcu_irq_enter_check_tick();

I suggest moving the WARN_ON_ONCE(in_nmi()) check here to avoid calling in_nmi() twice. Because of the READ_ONCE(), the compiler cannot (had better not!) eliminate the double call.

Makes sense.

Quoted 8 lines

kernel/rcu/tree.c void noinstr rcu_user_exit(void) · line 882

Quoted 5 lines
kernel/rcu/tree.c +5 -0
+void __rcu_irq_enter_check_tick(void)
+{
+	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
+
+	 // Enabling the tick is unsafe in NMI handlers.

There is an extra space before the "//", probably the one that used to be after the ";" below. ;-)

This is caused by my fundamental and not suppressible disgust of tail comments. They really disturb the reading flow for me.

          if (foo)
          	return; // Because ...

makes my pattern recognition stop because the semicolon is usually the end of the statement. But that's not the only reason.

         // Because ....
         if (foo)
         	return;

makes more sense to me because then the comment is explaining the condition and not the outcome. The outcome is obvious when the condition is well explained.

There are a few exceptions where I adjusted, e.g. in macros:

        foo();				\
        bar_or_something_else();	\

but only when the trailing backslash is properly aligned.

        foo();		\
        bar_or_something_else();	\

That stops the parser as well.

I know that this is a pet pieve but I can't help it to adjust it when I have a chance to do so :)

Quoted 10 lines

kernel/rcu/tree.c void noinstr rcu_user_exit(void) · line 887

Quoted 5 lines
kernel/rcu/tree.c +5 -0
+	if (WARN_ON_ONCE(in_nmi()))
+		return;
+
+	RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(),
+			 "Illegal rcu_irq_enter_check_tick() from extended quiescent state");

The instrumentation_begin() has disappeared, presumably because instrumentation is already enabled in the non-RCU code that directly calls rcu_irq_enter_check_tick(). (I do see the calls in rcu_nmi_enter() below.)

Yes. The intention here is to make sure that the caller does not misplace it. So if the call is in a non-instrumentable code path then objtool will complain and the developer will hopefully think twice whether this is the right place to wrap the call with instrumentation_* annotations. I know it's based on hope :)

Quoted 35 lines
Quoted 28 lines
kernel/rcu/tree.c +24 -0
+
+	if (!tick_nohz_full_cpu(rdp->cpu) ||
+	    !READ_ONCE(rdp->rcu_urgent_qs) ||
+	    READ_ONCE(rdp->rcu_forced_tick)) {
+		// RCU doesn't need nohz_full help from this CPU, or it is
+		// already getting that help.
+		return;
+	}
+
+	// We get here only when not in an extended quiescent state and
+	// from interrupts (as opposed to NMIs).  Therefore, (1) RCU is
+	// already watching and (2) The fact that we are in an interrupt
+	// handler and that the rcu_node lock is an irq-disabled lock
+	// prevents self-deadlock.  So we can safely recheck under the lock.
+	// Note that the nohz_full state currently cannot change.
+	raw_spin_lock_rcu_node(rdp->mynode);
+	if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
+		// A nohz_full CPU is in the kernel and RCU needs a
+		// quiescent state.  Turn on the tick!
+		WRITE_ONCE(rdp->rcu_forced_tick, true);
+		tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
+	}
+	raw_spin_unlock_rcu_node(rdp->mynode);
+}
 #endif /* CONFIG_NO_HZ_FULL */
 
 /**

kernel/rcu/tree.c noinstr void rcu_nmi_enter(void) · line 956

Quoted 2 lines
kernel/rcu/tree.c +0 -0
@@ -894,26 +955,7 @@ noinstr void rcu_nmi_enter(void)
 		incby = 1;
 	} else if (!in_nmi()) {

This can just be "else" given the in_nmi() check in __rcu_irq_enter_check_tick(), right? Ah, that check got a WARN_ON_ONCE(), so never mind!

Quoted 2 lines

I guess that will discourage NMI-handler calls to
rcu_irq_enter_check_tick(). ;-)

Exactly.

Quoted 5 lines

It does mean a double call to in_nmi(), though, so should that WARN_ON_ONCE(in_nmi()) check go into the rcu_irq_enter_check_tick() wrapper? Or do modern compilers figure this one out? Given the READ_ONCE() in preempt_count(), I have to say that I hope not. So see my comment above on rcu_irq_enter_check_tick().

Moving it to the wrapper is the right thing to do. Will fix.

Thanks,

tglx

#6 lore raw

Re: [patch V9 02/39] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()

In reply to Paul E. McKenney
Quoted 14 lines · Paul E. McKenney
Quoted 14 lines
(quoted diff) +14 -0
+	if (!tick_nohz_full_cpu(rdp->cpu) ||
+	    !READ_ONCE(rdp->rcu_urgent_qs) ||
+	    READ_ONCE(rdp->rcu_forced_tick)) {
+		// RCU doesn't need nohz_full help from this CPU, or it is
+		// already getting that help.
+		return;
+	}
+
+	// We get here only when not in an extended quiescent state and
+	// from interrupts (as opposed to NMIs).  Therefore, (1) RCU is
+	// already watching and (2) The fact that we are in an interrupt
+	// handler and that the rcu_node lock is an irq-disabled lock
+	// prevents self-deadlock.  So we can safely recheck under the lock.
+	// Note that the nohz_full state currently cannot change.
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+	raw_spin_lock_rcu_node(rdp->mynode);
+	if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
+		// A nohz_full CPU is in the kernel and RCU needs a
+		// quiescent state.  Turn on the tick!
+		WRITE_ONCE(rdp->rcu_forced_tick, true);
+		tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
+	}
+	raw_spin_unlock_rcu_node(rdp->mynode);

BTW., can we please not ever use this weird comment style in the future?

Linus gave an exception to single-line C++ style comments - but I don't think that should be extrapolated to a license to uglify the kernel with inconsistent muck like this. :-/

I've sanitized it via the patch below.

( I also fixed the whitespace damage and a capitalization typo while
at it, and fixed the spelling in the big comment explaining __rcu_irq_enter_check_tick(). )

Thanks,

Ingo

kernel/rcu/tree.c +20 -14
@@ -850,14 +850,14 @@ void noinstr rcu_user_exit(void)
 }
 
 /**
- * __rcu_irq_enter_check_tick - Enable scheduler tick on CPU if RCU needs it.
+ * __rcu_irq_enter_check_tick - Enable the scheduler tick on a CPU if RCU needs it.
  *
  * The scheduler tick is not normally enabled when CPUs enter the kernel
  * from nohz_full userspace execution.  After all, nohz_full userspace
  * execution is an RCU quiescent state and the time executing in the kernel
- * is quite short.  Except of course when it isn't.  And it is not hard to
+ * is quite short.  Except of course when it isn't: it is not hard to
  * cause a large system to spend tens of seconds or even minutes looping
- * in the kernel, which can cause a number of problems, include RCU CPU
+ * in the kernel, which can cause a number of problems, including RCU CPU
  * stall warnings.
  *
  * Therefore, if a nohz_full CPU fails to report a quiescent state
@@ -879,7 +879,7 @@ void __rcu_irq_enter_check_tick(void)
 {
 	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
 
-	 // Enabling the tick is unsafe in NMI handlers.
+	/* Enabling the tick is unsafe in NMI handlers. */
 	if (WARN_ON_ONCE(in_nmi()))
 		return;
 
@@ -889,21 +889,27 @@ void __rcu_irq_enter_check_tick(void)
 	if (!tick_nohz_full_cpu(rdp->cpu) ||
 	    !READ_ONCE(rdp->rcu_urgent_qs) ||
 	    READ_ONCE(rdp->rcu_forced_tick)) {
-		// RCU doesn't need nohz_full help from this CPU, or it is
-		// already getting that help.
+		/*
+		 * RCU doesn't need nohz_full help from this CPU, or it is
+		 * already getting that help.
+		 */
 		return;
 	}
 
-	// We get here only when not in an extended quiescent state and
-	// from interrupts (as opposed to NMIs).  Therefore, (1) RCU is
-	// already watching and (2) The fact that we are in an interrupt
-	// handler and that the rcu_node lock is an irq-disabled lock
-	// prevents self-deadlock.  So we can safely recheck under the lock.
-	// Note that the nohz_full state currently cannot change.
+	/*
+	 * We get here only when not in an extended quiescent state and
+	 * from interrupts (as opposed to NMIs).  Therefore, (1) RCU is
+	 * already watching and (2) the fact that we are in an interrupt
+	 * handler and that the rcu_node lock is an irq-disabled lock
+	 * prevents self-deadlock.  So we can safely recheck under the lock.
+	 * Note that the nohz_full state currently cannot change.
+	 */
 	raw_spin_lock_rcu_node(rdp->mynode);
 	if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
-		// A nohz_full CPU is in the kernel and RCU needs a
-		// quiescent state.  Turn on the tick!
+		/*
+		 * A nohz_full CPU is in the kernel and RCU needs a
+		 * quiescent state.  Turn on the tick!
+		 */
 		WRITE_ONCE(rdp->rcu_forced_tick, true);
 		tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
 	}
maintainer
#7 lore raw

Re: [patch V9 02/39] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()

Quoted 28 lines · Ingo Molnar
Quoted 14 lines · Paul E. McKenney
Quoted 14 lines
(quoted diff) +14 -0
+	if (!tick_nohz_full_cpu(rdp->cpu) ||
+	    !READ_ONCE(rdp->rcu_urgent_qs) ||
+	    READ_ONCE(rdp->rcu_forced_tick)) {
+		// RCU doesn't need nohz_full help from this CPU, or it is
+		// already getting that help.
+		return;
+	}
+
+	// We get here only when not in an extended quiescent state and
+	// from interrupts (as opposed to NMIs).  Therefore, (1) RCU is
+	// already watching and (2) The fact that we are in an interrupt
+	// handler and that the rcu_node lock is an irq-disabled lock
+	// prevents self-deadlock.  So we can safely recheck under the lock.
+	// Note that the nohz_full state currently cannot change.
Quoted 8 lines

Review

Quoted 8 lines
(quoted diff) +8 -0
+	raw_spin_lock_rcu_node(rdp->mynode);
+	if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
+		// A nohz_full CPU is in the kernel and RCU needs a
+		// quiescent state.  Turn on the tick!
+		WRITE_ONCE(rdp->rcu_forced_tick, true);
+		tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
+	}
+	raw_spin_unlock_rcu_node(rdp->mynode);

BTW., can we please not ever use this weird comment style in the future?

Quoted 5 lines

Linus gave an exception to single-line C++ style comments - but I don't think that should be extrapolated to a license to uglify the kernel with inconsistent muck like this. :-/

I've sanitized it via the patch below.

The "//" comment style does save vertical space. Is it really ugly or just unfamiliar? For purposes of comparison, back in the day, the "/* */" style seemed quite strange compared to my earlier languages' commenting styles.

Quoted 3 lines

( I also fixed the whitespace damage and a capitalization typo while
at it, and fixed the spelling in the big comment explaining __rcu_irq_enter_check_tick(). )

Some were stylistic rather than wrong, but I have no objection to any of these changes.

Thanx, Paul

Quoted 71 lines

Thanks,

Ingo

kernel/rcu/tree.c +20 -14
@@ -850,14 +850,14 @@ void noinstr rcu_user_exit(void)
 }
 
 /**
- * __rcu_irq_enter_check_tick - Enable scheduler tick on CPU if RCU needs it.
+ * __rcu_irq_enter_check_tick - Enable the scheduler tick on a CPU if RCU needs it.
  *
  * The scheduler tick is not normally enabled when CPUs enter the kernel
  * from nohz_full userspace execution.  After all, nohz_full userspace
  * execution is an RCU quiescent state and the time executing in the kernel
- * is quite short.  Except of course when it isn't.  And it is not hard to
+ * is quite short.  Except of course when it isn't: it is not hard to
  * cause a large system to spend tens of seconds or even minutes looping
- * in the kernel, which can cause a number of problems, include RCU CPU
+ * in the kernel, which can cause a number of problems, including RCU CPU
  * stall warnings.
  *
  * Therefore, if a nohz_full CPU fails to report a quiescent state
@@ -879,7 +879,7 @@ void __rcu_irq_enter_check_tick(void)
 {
 	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
 
-	 // Enabling the tick is unsafe in NMI handlers.
+	/* Enabling the tick is unsafe in NMI handlers. */
 	if (WARN_ON_ONCE(in_nmi()))
 		return;
 
@@ -889,21 +889,27 @@ void __rcu_irq_enter_check_tick(void)
 	if (!tick_nohz_full_cpu(rdp->cpu) ||
 	    !READ_ONCE(rdp->rcu_urgent_qs) ||
 	    READ_ONCE(rdp->rcu_forced_tick)) {
-		// RCU doesn't need nohz_full help from this CPU, or it is
-		// already getting that help.
+		/*
+		 * RCU doesn't need nohz_full help from this CPU, or it is
+		 * already getting that help.
+		 */
 		return;
 	}
 
-	// We get here only when not in an extended quiescent state and
-	// from interrupts (as opposed to NMIs).  Therefore, (1) RCU is
-	// already watching and (2) The fact that we are in an interrupt
-	// handler and that the rcu_node lock is an irq-disabled lock
-	// prevents self-deadlock.  So we can safely recheck under the lock.
-	// Note that the nohz_full state currently cannot change.
+	/*
+	 * We get here only when not in an extended quiescent state and
+	 * from interrupts (as opposed to NMIs).  Therefore, (1) RCU is
+	 * already watching and (2) the fact that we are in an interrupt
+	 * handler and that the rcu_node lock is an irq-disabled lock
+	 * prevents self-deadlock.  So we can safely recheck under the lock.
+	 * Note that the nohz_full state currently cannot change.
+	 */
 	raw_spin_lock_rcu_node(rdp->mynode);
 	if (rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
-		// A nohz_full CPU is in the kernel and RCU needs a
-		// quiescent state.  Turn on the tick!
+		/*
+		 * A nohz_full CPU is in the kernel and RCU needs a
+		 * quiescent state.  Turn on the tick!
+		 */
 		WRITE_ONCE(rdp->rcu_forced_tick, true);
 		tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
 	}
#8 lore raw

[tip: x86/entry] rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()

In reply to Thomas Gleixner
The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     aaf2bc50df1f4bfc6857fc601fc7b21d5a18c6a1
Gitweb:        https://git.kernel.org/tip/aaf2bc50df1f4bfc6857fc601fc7b21d5a18c6a1
Author:        Paul E. McKenney <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:15 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:04:18 +02:00

rcu: Abstract out rcu_irq_enter_check_tick() from rcu_nmi_enter()

There will likely be exception handlers that can sleep, which rules out the usual approach of invoking rcu_nmi_enter() on entry and also rcu_nmi_exit() on all exit paths. However, the alternative approach of just not calling anything can prevent RCU from coaxing quiescent states from nohz_full CPUs that are looping in the kernel: RCU must instead IPI them explicitly. It would be better to enable the scheduler tick on such CPUs to interact with RCU in a lighter-weight manner, and this enabling is one of the things that rcu_nmi_enter() currently does.

What is needed is something that helps RCU coax quiescent states while not preventing subsequent sleeps. This commit therefore splits out the nohz_full scheduler-tick enabling from the rest of the rcu_nmi_enter() logic into a new function named rcu_irq_enter_check_tick().

[ tglx: Renamed the function and made it a nop when context tracking is off ] [ mingo: Fixed a CONFIG_NO_HZ_FULL assumption, harmonized and fixed all the
comment blocks and cleaned up rcu_nmi_enter()/exit() definitions. ]

---
 include/linux/hardirq.h | 29 ++++++++------
 kernel/rcu/tree.c       | 82 ++++++++++++++++++++++++++++++----------
 2 files changed, 79 insertions(+), 32 deletions(-)
include/linux/hardirq.h +17 -12
index 621556e..e07cf85 100644
@@ -2,31 +2,28 @@
 #ifndef LINUX_HARDIRQ_H
 #define LINUX_HARDIRQ_H
 
+#include <linux/context_tracking_state.h>
 #include <linux/preempt.h>
 #include <linux/lockdep.h>
 #include <linux/ftrace_irq.h>
 #include <linux/vtime.h>
 #include <asm/hardirq.h>
 
-
 extern void synchronize_irq(unsigned int irq);
 extern bool synchronize_hardirq(unsigned int irq);
 
-#if defined(CONFIG_TINY_RCU)
-
-static inline void rcu_nmi_enter(void)
-{
-}
+#ifdef CONFIG_NO_HZ_FULL
+void __rcu_irq_enter_check_tick(void);
+#else
+static inline void __rcu_irq_enter_check_tick(void) { }
+#endif
 
-static inline void rcu_nmi_exit(void)
+static __always_inline void rcu_irq_enter_check_tick(void)
 {
+	if (context_tracking_enabled())
+		__rcu_irq_enter_check_tick();
 }
 
-#else
-extern void rcu_nmi_enter(void);
-extern void rcu_nmi_exit(void);
-#endif
-
 /*
  * It is safe to do non-atomic ops on ->hardirq_context,
  * because NMI handlers may not preempt and the ops are
@@ -65,6 +62,14 @@ extern void irq_exit(void);
 #define arch_nmi_exit()		do { } while (0)
 #endif
 
+#ifdef CONFIG_TINY_RCU
+static inline void rcu_nmi_enter(void) { }
+static inline void rcu_nmi_exit(void) { }
+#else
+extern void rcu_nmi_enter(void);
+extern void rcu_nmi_exit(void);
+#endif
+
 /*
  * NMI vs Tracing
  * --------------
kernel/rcu/tree.c +62 -20

94 lines not on this page · raw message

#9 lore raw

PATCH v9 3/39 rcu: Provide rcu_irq_exit_check_preempt()

In reply to Thomas Gleixner

Provide a debug check which can be invoked from exception return to kernel mode before an attempt is made to schedule. Warn if RCU is not ready for this.

---
V9: New patch

---
 include/linux/rcutiny.h |    1 +
 include/linux/rcutree.h |    6 ++++++
 kernel/rcu/tree.c       |   18 ++++++++++++++++++
 3 files changed, 25 insertions(+)
include/linux/rcutiny.h +1 -0
@@ -72,6 +72,7 @@ static inline void rcu_irq_exit_irqson(v
 static inline void rcu_irq_enter_irqson(void) { }
 static inline void rcu_irq_exit(void) { }
 static inline void rcu_irq_exit_preempt(void) { }
+static inline void rcu_irq_exit_check_preempt(void) { }
 static inline void exit_rcu(void) { }
 static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)
 {
include/linux/rcutree.h +6 -0
@@ -51,6 +51,12 @@ void rcu_irq_exit_preempt(void);
 void rcu_irq_enter_irqson(void);
 void rcu_irq_exit_irqson(void);
 
+#ifdef CONFIG_PROVE_RCU
+void rcu_irq_exit_check_preempt(void);
+#else
+static inline void rcu_irq_exit_check_preempt(void) { }
+#endif
+
 void exit_rcu(void);
 
 void rcu_scheduler_starting(void);
kernel/rcu/tree.c +18 -0
@@ -765,6 +765,24 @@ void rcu_irq_exit_preempt(void)
 			 "RCU in extended quiescent state!");
 }
 
+#ifdef CONFIG_PROVE_RCU
+/**
+ * rcu_irq_exit_check_preempt - Validate that scheduling is possible
+ */
+void rcu_irq_exit_check_preempt(void)
+{
+	lockdep_assert_irqs_disabled();
+
+	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) <= 0,
+			 "RCU dynticks_nesting counter underflow/zero!");
+	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) !=
+			 DYNTICK_IRQ_NONIDLE,
+			 "Bad RCU  dynticks_nmi_nesting counter\n");
+	RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(),
+			 "RCU in extended quiescent state!");
+}
+#endif /* #ifdef CONFIG_PROVE_RCU */
+
 /*
  * Wrapper for rcu_irq_exit() where interrupts are enabled.
  *
#10 lore raw

[tip: x86/entry] rcu: Provide rcu_irq_exit_check_preempt()

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     07325d4a90d2d84de45cc07b134fd0f023dbb971
Gitweb:        https://git.kernel.org/tip/07325d4a90d2d84de45cc07b134fd0f023dbb971
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:16 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:05:11 +02:00

rcu: Provide rcu_irq_exit_check_preempt()

Provide a debug check which can be invoked from exception return to kernel mode before an attempt is made to schedule. Warn if RCU is not ready for this.

---
 include/linux/rcutiny.h |  1 +
 include/linux/rcutree.h |  6 ++++++
 kernel/rcu/tree.c       | 18 ++++++++++++++++++
 3 files changed, 25 insertions(+)
include/linux/rcutiny.h +1 -0
index c869fb2..8512cae 100644
@@ -72,6 +72,7 @@ static inline void rcu_irq_exit_irqson(void) { }
 static inline void rcu_irq_enter_irqson(void) { }
 static inline void rcu_irq_exit(void) { }
 static inline void rcu_irq_exit_preempt(void) { }
+static inline void rcu_irq_exit_check_preempt(void) { }
 static inline void exit_rcu(void) { }
 static inline bool rcu_preempt_need_deferred_qs(struct task_struct *t)
 {
include/linux/rcutree.h +6 -0
index 9366fa4..d5cc9d6 100644
@@ -51,6 +51,12 @@ void rcu_irq_exit_preempt(void);
 void rcu_irq_enter_irqson(void);
 void rcu_irq_exit_irqson(void);
 
+#ifdef CONFIG_PROVE_RCU
+void rcu_irq_exit_check_preempt(void);
+#else
+static inline void rcu_irq_exit_check_preempt(void) { }
+#endif
+
 void exit_rcu(void);
 
 void rcu_scheduler_starting(void);
kernel/rcu/tree.c +18 -0
index b7f8c49..d8e9dbb 100644
@@ -765,6 +765,24 @@ void rcu_irq_exit_preempt(void)
 			 "RCU in extended quiescent state!");
 }
 
+#ifdef CONFIG_PROVE_RCU
+/**
+ * rcu_irq_exit_check_preempt - Validate that scheduling is possible
+ */
+void rcu_irq_exit_check_preempt(void)
+{
+	lockdep_assert_irqs_disabled();
+
+	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) <= 0,
+			 "RCU dynticks_nesting counter underflow/zero!");
+	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) !=
+			 DYNTICK_IRQ_NONIDLE,
+			 "Bad RCU  dynticks_nmi_nesting counter\n");
+	RCU_LOCKDEP_WARN(rcu_dynticks_curr_cpu_in_eqs(),
+			 "RCU in extended quiescent state!");
+}
+#endif /* #ifdef CONFIG_PROVE_RCU */
+
 /*
  * Wrapper for rcu_irq_exit() where interrupts are enabled.
  *
#11 lore raw

PATCH v9 4/39 x86/entry: Provide idtentry_entry/exit_cond_rcu()

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

After a lengthy discussion [1] it turned out that RCU does not need a full rcu_irq_enter/exit() when RCU is already watching. All it needs if NOHZ_FULL is active is to check whether the tick needs to be restarted.

This allows to avoid a separate variant for the pagefault handler which cannot invoke rcu_irq_enter() on a kernel pagefault which might sleep.

The cond_rcu argument is only temporary and will be removed once the existing users of idtentry_enter/exit() have been cleaned up. After that the code can be significantly simplified.

---
V9: Reworked to the new RCU mode

---
 arch/x86/entry/common.c         |   84 ++++++++++++++++++++++++++++++----------
 arch/x86/include/asm/idtentry.h |   14 +++++-
 2 files changed, 77 insertions(+), 21 deletions(-)
arch/x86/entry/common.c +65 -19
@@ -512,8 +512,10 @@ SYSCALL_DEFINE0(ni_syscall)
 }
 
 /**
- * idtentry_enter - Handle state tracking on idtentry
+ * idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional
+ *			     RCU handling
  * @regs:	Pointer to pt_regs of interrupted context
+ * @cond_rcu:	Invoke rcu_irq_enter() only if RCU is not watching
  *
  * Invokes:
  *  - lockdep irqflag state tracking as low level ASM entry disabled
@@ -521,40 +523,81 @@ SYSCALL_DEFINE0(ni_syscall)
  *
  *  - Context tracking if the exception hit user mode.
  *
- *  - RCU notification if the exception hit kernel mode.
- *
  *  - The hardirq tracer to keep the state consistent as low level ASM
  *    entry disabled interrupts.
+ *
+ * For kernel mode entries RCU handling is done conditional. If RCU is
+ * watching then the only RCU requirement is to check whether the tick has
+ * to be restarted. If RCU is not watching then rcu_irq_enter() has to be
+ * invoked on entry and rcu_irq_exit() on exit.
+ *
+ * Avoiding the rcu_irq_enter/exit() calls is an optimization but also
+ * solves the problem of kernel mode pagefaults which can schedule, which
+ * is not possible after invoking rcu_irq_enter() without undoing it.
+ *
+ * For user mode entries enter_from_user_mode() must be invoked to
+ * establish the proper context for NOHZ_FULL. Otherwise scheduling on exit
+ * would not be possible.
+ *
+ * Returns: True if RCU has been adjusted on a kernel entry
+ *	    False otherwise
+ *
+ * The return value must be fed into the rcu_exit argument of
+ * idtentry_exit_cond_rcu().
  */
-void noinstr idtentry_enter(struct pt_regs *regs)
+bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu)
 {
 	if (user_mode(regs)) {
 		enter_from_user_mode();
 	} else {
-		lockdep_hardirqs_off(CALLER_ADDR0);
-		rcu_irq_enter();
-		instrumentation_begin();
-		trace_hardirqs_off_prepare();
-		instrumentation_end();
+		if (!cond_rcu || !__rcu_is_watching()) {
+			/*
+			 * If RCU is not watching then the same careful
+			 * sequence vs. lockdep and tracing is required
+			 * as in enter_from_user_mode().
+			 *
+			 * This only happens for IRQs that hits the idle
+			 * loop, i.e. if idle is not using MWAIT.
+			 */
+			lockdep_hardirqs_off(CALLER_ADDR0);
+			rcu_irq_enter();
+			instrumentation_begin();
+			trace_hardirqs_off_prepare();
+			instrumentation_end();
+			return true;
+		} else {
+			/*
+			 * If RCU is watching then RCU only wants to check
+			 * whether it needs to restart the tick in NOHZ
+			 * mode.
+			 */
+			instrumentation_begin();
+			rcu_irq_enter_check_tick();
+			/* Use the combo lockdep/tracing function */
+			trace_hardirqs_off();
+			instrumentation_end();
+		}
 	}
+	return false;
 }
 
 /**
- * idtentry_exit - Common code to handle return from exceptions
+ * idtentry_exit_cond_rcu - Handle return from exception with conditional RCU
+ *			    handling
  * @regs:	Pointer to pt_regs (exception entry regs)
+ * @rcu_exit:	Invoke rcu_irq_exit() if true
  *
  * Depending on the return target (kernel/user) this runs the necessary
- * preemption and work checks if possible and required and returns to
+ * preemption and work checks if possible and reguired and returns to
  * the caller with interrupts disabled and no further work pending.
  *
  * This is the last action before returning to the low level ASM code which
  * just needs to return to the appropriate context.
  *
- * Invoked by all exception/interrupt IDTENTRY handlers which are not
- * returning through the paranoid exit path (all except NMI, #DF and the IST
- * variants of #MC and #DB) and are therefore on the thread stack.
+ * Counterpart to idtentry_enter_cond_rcu(). The return value of the entry
+ * function must be fed into the @rcu_exit argument.
  */
-void noinstr idtentry_exit(struct pt_regs *regs)
+void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit)
 {
 	lockdep_assert_irqs_disabled();
 
@@ -580,7 +623,8 @@ void noinstr idtentry_exit(struct pt_reg
 				if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
 					WARN_ON_ONCE(!on_thread_stack());
 				instrumentation_begin();
-				rcu_irq_exit_preempt();
+				if (rcu_exit)
+					rcu_irq_exit_preempt();
 				if (need_resched())
 					preempt_schedule_irq();
 				/* Covers both tracing and lockdep */
@@ -602,10 +646,12 @@ void noinstr idtentry_exit(struct pt_reg
 		trace_hardirqs_on_prepare();
 		lockdep_hardirqs_on_prepare(CALLER_ADDR0);
 		instrumentation_end();
-		rcu_irq_exit();
+		if (rcu_exit)
+			rcu_irq_exit();
 		lockdep_hardirqs_on(CALLER_ADDR0);
 	} else {
-		/* IRQ flags state is correct already. Just tell RCU */
-		rcu_irq_exit();
+		/* IRQ flags state is correct already. Just tell RCU. */
+		if (rcu_exit)
+			rcu_irq_exit();
 	}
 }
arch/x86/include/asm/idtentry.h +12 -2
@@ -7,8 +7,18 @@
 
 #ifndef __ASSEMBLY__
 
-void idtentry_enter(struct pt_regs *regs);
-void idtentry_exit(struct pt_regs *regs);
+bool idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu);
+void idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit);
+
+static __always_inline void idtentry_enter(struct pt_regs *regs)
+{
+	idtentry_enter_cond_rcu(regs, false);
+}
+
+static __always_inline void idtentry_exit(struct pt_regs *regs)
+{
+	idtentry_exit_cond_rcu(regs, true);
+}
 
 /**
  * DECLARE_IDTENTRY - Declare functions for simple IDT entry points
maintainer
#12 lore raw

Re: [patch V9 04/39] x86/entry: Provide idtentry_entry/exit_cond_rcu()

Quoted 6 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>

After a lengthy discussion [1] it turned out that RCU does not need a full rcu_irq_enter/exit() when RCU is already watching. All it needs if NOHZ_FULL is active is to check whether the tick needs to be restarted.

Quoted 6 lines

This allows to avoid a separate variant for the pagefault handler which cannot invoke rcu_irq_enter() on a kernel pagefault which might sleep.

The cond_rcu argument is only temporary and will be removed once the existing users of idtentry_enter/exit() have been cleaned up. After that the code can be significantly simplified.

And yes, I see that the sole call to rcu_irq_enter_check_tick() does have instrumentation turned on, so good! ;-)

Thanx, Paul

#13 lore raw

Re: [patch V9 04/39] x86/entry: Provide idtentry_entry/exit_cond_rcu()

In reply to Thomas Gleixner
Quoted 95 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>

After a lengthy discussion [1] it turned out that RCU does not need a full rcu_irq_enter/exit() when RCU is already watching. All it needs if NOHZ_FULL is active is to check whether the tick needs to be restarted.

This allows to avoid a separate variant for the pagefault handler which cannot invoke rcu_irq_enter() on a kernel pagefault which might sleep.

The cond_rcu argument is only temporary and will be removed once the existing users of idtentry_enter/exit() have been cleaned up. After that the code can be significantly simplified.

---
V9: Reworked to the new RCU mode

---
 arch/x86/entry/common.c         |   84 ++++++++++++++++++++++++++++++----------
 arch/x86/include/asm/idtentry.h |   14 +++++-
 2 files changed, 77 insertions(+), 21 deletions(-)
arch/x86/entry/common.c +43 -9
@@ -512,8 +512,10 @@ SYSCALL_DEFINE0(ni_syscall)
 }
 
 /**
- * idtentry_enter - Handle state tracking on idtentry
+ * idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional
+ *			     RCU handling
  * @regs:	Pointer to pt_regs of interrupted context
+ * @cond_rcu:	Invoke rcu_irq_enter() only if RCU is not watching
  *
  * Invokes:
  *  - lockdep irqflag state tracking as low level ASM entry disabled
@@ -521,40 +523,81 @@ SYSCALL_DEFINE0(ni_syscall)
  *
  *  - Context tracking if the exception hit user mode.
  *
- *  - RCU notification if the exception hit kernel mode.
- *
  *  - The hardirq tracer to keep the state consistent as low level ASM
  *    entry disabled interrupts.
+ *
+ * For kernel mode entries RCU handling is done conditional. If RCU is
+ * watching then the only RCU requirement is to check whether the tick has
+ * to be restarted. If RCU is not watching then rcu_irq_enter() has to be
+ * invoked on entry and rcu_irq_exit() on exit.
+ *
+ * Avoiding the rcu_irq_enter/exit() calls is an optimization but also
+ * solves the problem of kernel mode pagefaults which can schedule, which
+ * is not possible after invoking rcu_irq_enter() without undoing it.
+ *
+ * For user mode entries enter_from_user_mode() must be invoked to
+ * establish the proper context for NOHZ_FULL. Otherwise scheduling on exit
+ * would not be possible.
+ *
+ * Returns: True if RCU has been adjusted on a kernel entry
+ *	    False otherwise
+ *
+ * The return value must be fed into the rcu_exit argument of
+ * idtentry_exit_cond_rcu().
  */
-void noinstr idtentry_enter(struct pt_regs *regs)
+bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu)
 {
 	if (user_mode(regs)) {
 		enter_from_user_mode();
 	} else {
-		lockdep_hardirqs_off(CALLER_ADDR0);
-		rcu_irq_enter();
-		instrumentation_begin();
-		trace_hardirqs_off_prepare();
-		instrumentation_end();
+		if (!cond_rcu || !__rcu_is_watching()) {
+			/*
+			 * If RCU is not watching then the same careful
+			 * sequence vs. lockdep and tracing is required
+			 * as in enter_from_user_mode().
+			 *
+			 * This only happens for IRQs that hits the idle
+			 * loop, i.e. if idle is not using MWAIT.
+			 */
+			lockdep_hardirqs_off(CALLER_ADDR0);
+			rcu_irq_enter();
+			instrumentation_begin();
+			trace_hardirqs_off_prepare();
+			instrumentation_end();
+			return true;
+		} else {
+			/*
+			 * If RCU is watching then RCU only wants to check
+			 * whether it needs to restart the tick in NOHZ
+			 * mode.

arch/x86/entry/common.c SYSCALL_DEFINE0(ni_syscall) · line 582

Quoted 10 lines
arch/x86/entry/common.c +8 -0
+			 */
+			instrumentation_begin();
+			rcu_irq_enter_check_tick();
+			/* Use the combo lockdep/tracing function */
+			trace_hardirqs_off();
+			instrumentation_end();
+		}
 	}
+	return false;
 }

Any objections to the simplified/flattened control flow below?

Thanks,

Ingo

 common.c |   59 +++++++++++++++++++++++++++++++----------------------------
 1 file changed, 31 insertions(+), 28 deletions(-)
arch/x86/entry/common.c +31 -28
Index: tip/arch/x86/entry/common.c
===================================================================
@@ -549,35 +549,38 @@ bool noinstr idtentry_enter_cond_rcu(str
 {
 	if (user_mode(regs)) {
 		enter_from_user_mode();
-	} else {
-		if (!cond_rcu || !__rcu_is_watching()) {
-			/*
-			 * If RCU is not watching then the same careful
-			 * sequence vs. lockdep and tracing is required
-			 * as in enter_from_user_mode().
-			 *
-			 * This only happens for IRQs that hit the idle
-			 * loop, i.e. if idle is not using MWAIT.
-			 */
-			lockdep_hardirqs_off(CALLER_ADDR0);
-			rcu_irq_enter();
-			instrumentation_begin();
-			trace_hardirqs_off_prepare();
-			instrumentation_end();
-			return true;
-		} else {
-			/*
-			 * If RCU is watching then RCU only wants to check
-			 * whether it needs to restart the tick in NOHZ
-			 * mode.
-			 */
-			instrumentation_begin();
-			rcu_irq_enter_check_tick();
-			/* Use the combo lockdep/tracing function */
-			trace_hardirqs_off();
-			instrumentation_end();
-		}
+		return false;
 	}
+
+	if (!cond_rcu || !__rcu_is_watching()) {
+		/*
+		 * If RCU is not watching then the same careful
+		 * sequence vs. lockdep and tracing is required
+		 * as in enter_from_user_mode().
+		 *
+		 * This only happens for IRQs that hit the idle
+		 * loop, i.e. if idle is not using MWAIT.
+		 */
+		lockdep_hardirqs_off(CALLER_ADDR0);
+		rcu_irq_enter();
+		instrumentation_begin();
+		trace_hardirqs_off_prepare();
+		instrumentation_end();
+
+		return true;
+	}
+
+	/*
+	 * If RCU is watching then RCU only wants to check
+	 * whether it needs to restart the tick in NOHZ
+	 * mode.
+	 */
+	instrumentation_begin();
+	rcu_irq_enter_check_tick();
+	/* Use the combo lockdep/tracing function */
+	trace_hardirqs_off();
+	instrumentation_end();
+
 	return false;
 }
 
#14 lore raw

Re: [patch V9 04/39] x86/entry: Provide idtentry_entry/exit_cond_rcu()

Quoted 3 lines · Ingo Molnar

* Thomas Gleixner <[email protected]> wrote:

Any objections to the simplified/flattened control flow below?

No, but you really don't want to do it at this point. Patch 7/39 is where you want to do that.

#15 lore raw

PATCH v9 5/39 x86/entry: Provide idtentry_enter/exit_user()

In reply to Thomas Gleixner

As there are exceptions which already handle entry from user mode and from kernel mode separately having an explicit user handling makes sense and makes the code easier to understand.

---
V9: New patch

---
 arch/x86/entry/common.c         |   31 +++++++++++++++++++++++++++++++
 arch/x86/include/asm/idtentry.h |    3 +++
 2 files changed, 34 insertions(+)
arch/x86/entry/common.c +31 -0
@@ -655,3 +655,34 @@ void noinstr idtentry_exit_cond_rcu(stru
 			rcu_irq_exit();
 	}
 }
+
+/**
+ * idtentry_enter_user - Handle state tracking on idtentry from user mode
+ * @regs:	Pointer to pt_regs of interrupted context
+ *
+ * Invokes enter_from_user_mode() to establish the proper context for
+ * NOHZ_FULL. Otherwise scheduling on exit would not be possible.
+ */
+void noinstr idtentry_enter_user(struct pt_regs *regs)
+{
+	enter_from_user_mode();
+}
+
+/**
+ * idtentry_exit_user - Handle return from exception to user mode
+ * @regs:	Pointer to pt_regs (exception entry regs)
+ *
+ * Runs the necessary preemption and work checks and returns to the caller
+ * with interrupts disabled and no further work pending.
+ *
+ * This is the last action before returning to the low level ASM code which
+ * just needs to return to the appropriate context.
+ *
+ * Counterpart to idtentry_enter_user().
+ */
+void noinstr idtentry_exit_user(struct pt_regs *regs)
+{
+	lockdep_assert_irqs_disabled();
+
+	prepare_exit_to_usermode(regs);
+}
arch/x86/include/asm/idtentry.h +3 -0
@@ -7,6 +7,9 @@
 
 #ifndef __ASSEMBLY__
 
+void idtentry_enter_user(struct pt_regs *regs);
+void idtentry_exit_user(struct pt_regs *regs);
+
 bool idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu);
 void idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit);
 
#16 lore raw

[tip: x86/entry] x86/entry: Provide idtentry_enter/exit_user()

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     d4b1f51731fd10a836e46f54c96a8913d945735b
Gitweb:        https://git.kernel.org/tip/d4b1f51731fd10a836e46f54c96a8913d945735b
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:18 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

x86/entry: Provide idtentry_enter/exit_user()

As there are exceptions which already handle entry from user mode and from kernel mode separately, providing explicit user entry/exit handling callbacks makes sense and makes the code easier to understand.

---
 arch/x86/entry/common.c         | 31 +++++++++++++++++++++++++++++++
 arch/x86/include/asm/idtentry.h |  3 +++
 2 files changed, 34 insertions(+)
arch/x86/entry/common.c +31 -0
index a7f5846..b7fcb13 100644
@@ -658,3 +658,34 @@ void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit)
 			rcu_irq_exit();
 	}
 }
+
+/**
+ * idtentry_enter_user - Handle state tracking on idtentry from user mode
+ * @regs:	Pointer to pt_regs of interrupted context
+ *
+ * Invokes enter_from_user_mode() to establish the proper context for
+ * NOHZ_FULL. Otherwise scheduling on exit would not be possible.
+ */
+void noinstr idtentry_enter_user(struct pt_regs *regs)
+{
+	enter_from_user_mode();
+}
+
+/**
+ * idtentry_exit_user - Handle return from exception to user mode
+ * @regs:	Pointer to pt_regs (exception entry regs)
+ *
+ * Runs the necessary preemption and work checks and returns to the caller
+ * with interrupts disabled and no further work pending.
+ *
+ * This is the last action before returning to the low level ASM code which
+ * just needs to return to the appropriate context.
+ *
+ * Counterpart to idtentry_enter_user().
+ */
+void noinstr idtentry_exit_user(struct pt_regs *regs)
+{
+	lockdep_assert_irqs_disabled();
+
+	prepare_exit_to_usermode(regs);
+}
arch/x86/include/asm/idtentry.h +3 -0
index a116b80..b3aca72 100644
@@ -7,6 +7,9 @@
 
 #ifndef __ASSEMBLY__
 
+void idtentry_enter_user(struct pt_regs *regs);
+void idtentry_exit_user(struct pt_regs *regs);
+
 bool idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu);
 void idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit);
 
#17 lore raw

PATCH v9 6/39 x86/idtentry: Switch to conditional RCU handling

In reply to Thomas Gleixner

Switch all idtentry_enter/exit() users over to the new conditional RCU handling scheme and make the user mode entries in #DB, #INT3 and #MCE use the user mode idtentry functions.

---
V9: New patch

---
 arch/x86/include/asm/idtentry.h |   10 ++++++----
 arch/x86/kernel/cpu/mce/core.c  |    4 ++--
 arch/x86/kernel/traps.c         |   10 +++++-----
 3 files changed, 13 insertions(+), 11 deletions(-)
arch/x86/include/asm/idtentry.h +6 -4
@@ -61,11 +61,12 @@ static __always_inline void __##func(str
 									\
 __visible noinstr void func(struct pt_regs *regs)			\
 {									\
-	idtentry_enter(regs);						\
+	bool rcu_exit = idtentry_enter_cond_rcu(regs);			\
+									\
 	instrumentation_begin();					\
 	__##func (regs);						\
 	instrumentation_end();						\
-	idtentry_exit(regs);						\
+	idtentry_exit_cond_rcu(regs, rcu_exit);				\
 }									\
 									\
 static __always_inline void __##func(struct pt_regs *regs)
@@ -107,11 +108,12 @@ static __always_inline void __##func(str
 __visible noinstr void func(struct pt_regs *regs,			\
 			    unsigned long error_code)			\
 {									\
-	idtentry_enter(regs);						\
+	bool rcu_exit = idtentry_enter_cond_rcu(regs);			\
+									\
 	instrumentation_begin();					\
 	__##func (regs, error_code);					\
 	instrumentation_end();						\
-	idtentry_exit(regs);						\
+	idtentry_exit_cond_rcu(regs, rcu_exit);				\
 }									\
 									\
 static __always_inline void __##func(struct pt_regs *regs,		\
arch/x86/kernel/cpu/mce/core.c +2 -2
@@ -1929,11 +1929,11 @@ static __always_inline void exc_machine_
 
 static __always_inline void exc_machine_check_user(struct pt_regs *regs)
 {
-	idtentry_enter(regs);
+	idtentry_enter_user(regs);
 	instrumentation_begin();
 	machine_check_vector(regs);
 	instrumentation_end();
-	idtentry_exit(regs);
+	idtentry_exit_user(regs);
 }
 
 #ifdef CONFIG_X86_64
arch/x86/kernel/traps.c +5 -5
@@ -619,18 +619,18 @@ DEFINE_IDTENTRY_RAW(exc_int3)
 		return;
 
 	/*
-	 * idtentry_enter() uses static_branch_{,un}likely() and therefore
+	 * idtentry_enter_user() uses static_branch_{,un}likely() and therefore
 	 * can trigger INT3, hence poke_int3_handler() must be done
 	 * before. If the entry came from kernel mode, then use nmi_enter()
 	 * because the INT3 could have been hit in any context including
 	 * NMI.
 	 */
 	if (user_mode(regs)) {
-		idtentry_enter(regs);
+		idtentry_enter_user(regs);
 		instrumentation_begin();
 		do_int3_user(regs);
 		instrumentation_end();
-		idtentry_exit(regs);
+		idtentry_exit_user(regs);
 	} else {
 		nmi_enter();
 		instrumentation_begin();
@@ -877,7 +877,7 @@ static __always_inline void exc_debug_ke
 static __always_inline void exc_debug_user(struct pt_regs *regs,
 					   unsigned long dr6)
 {
-	idtentry_enter(regs);
+	idtentry_enter_user(regs);
 	clear_thread_flag(TIF_BLOCKSTEP);
 
 	/*
@@ -886,7 +886,7 @@ static __always_inline void exc_debug_us
 	 * User wants a sigtrap for that.
 	 */
 	handle_debug(regs, dr6, !dr6);
-	idtentry_exit(regs);
+	idtentry_exit_user(regs);
 }
 
 #ifdef CONFIG_X86_64
#18 lore raw

[tip: x86/entry] x86/idtentry: Switch to conditional RCU handling

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     77d7843297dfe3bbd997a46548c9cdd8047ecffe
Gitweb:        https://git.kernel.org/tip/77d7843297dfe3bbd997a46548c9cdd8047ecffe
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:19 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

x86/idtentry: Switch to conditional RCU handling

Switch all idtentry_enter/exit() users over to the new conditional RCU handling scheme and make the user mode entries in #DB, #INT3 and #MCE use the user mode idtentry functions.

---
 arch/x86/include/asm/idtentry.h | 10 ++++++----
 arch/x86/kernel/cpu/mce/core.c  |  4 ++--
 arch/x86/kernel/traps.c         | 10 +++++-----
 3 files changed, 13 insertions(+), 11 deletions(-)
arch/x86/include/asm/idtentry.h +6 -4
index b3aca72..0f974e5 100644
@@ -61,11 +61,12 @@ static __always_inline void __##func(struct pt_regs *regs);		\
 									\
 __visible noinstr void func(struct pt_regs *regs)			\
 {									\
-	idtentry_enter(regs);						\
+	bool rcu_exit = idtentry_enter_cond_rcu(regs);			\
+									\
 	instrumentation_begin();					\
 	__##func (regs);						\
 	instrumentation_end();						\
-	idtentry_exit(regs);						\
+	idtentry_exit_cond_rcu(regs, rcu_exit);				\
 }									\
 									\
 static __always_inline void __##func(struct pt_regs *regs)
@@ -107,11 +108,12 @@ static __always_inline void __##func(struct pt_regs *regs,		\
 __visible noinstr void func(struct pt_regs *regs,			\
 			    unsigned long error_code)			\
 {									\
-	idtentry_enter(regs);						\
+	bool rcu_exit = idtentry_enter_cond_rcu(regs);			\
+									\
 	instrumentation_begin();					\
 	__##func (regs, error_code);					\
 	instrumentation_end();						\
-	idtentry_exit(regs);						\
+	idtentry_exit_cond_rcu(regs, rcu_exit);				\
 }									\
 									\
 static __always_inline void __##func(struct pt_regs *regs,		\
arch/x86/kernel/cpu/mce/core.c +2 -2
index a32a7e2..c47f004 100644
@@ -1929,11 +1929,11 @@ static __always_inline void exc_machine_check_kernel(struct pt_regs *regs)
 
 static __always_inline void exc_machine_check_user(struct pt_regs *regs)
 {
-	idtentry_enter(regs);
+	idtentry_enter_user(regs);
 	instrumentation_begin();
 	machine_check_vector(regs);
 	instrumentation_end();
-	idtentry_exit(regs);
+	idtentry_exit_user(regs);
 }
 
 #ifdef CONFIG_X86_64
arch/x86/kernel/traps.c +5 -5
index 9e5d81c..f28be3e 100644
@@ -619,18 +619,18 @@ DEFINE_IDTENTRY_RAW(exc_int3)
 		return;
 
 	/*
-	 * idtentry_enter() uses static_branch_{,un}likely() and therefore
+	 * idtentry_enter_user() uses static_branch_{,un}likely() and therefore
 	 * can trigger INT3, hence poke_int3_handler() must be done
 	 * before. If the entry came from kernel mode, then use nmi_enter()
 	 * because the INT3 could have been hit in any context including
 	 * NMI.
 	 */
 	if (user_mode(regs)) {
-		idtentry_enter(regs);
+		idtentry_enter_user(regs);
 		instrumentation_begin();
 		do_int3_user(regs);
 		instrumentation_end();
-		idtentry_exit(regs);
+		idtentry_exit_user(regs);
 	} else {
 		nmi_enter();
 		instrumentation_begin();
@@ -877,7 +877,7 @@ static __always_inline void exc_debug_kernel(struct pt_regs *regs,
 static __always_inline void exc_debug_user(struct pt_regs *regs,
 					   unsigned long dr6)
 {
-	idtentry_enter(regs);
+	idtentry_enter_user(regs);
 	clear_thread_flag(TIF_BLOCKSTEP);
 
 	/*
@@ -886,7 +886,7 @@ static __always_inline void exc_debug_user(struct pt_regs *regs,
 	 * User wants a sigtrap for that.
 	 */
 	handle_debug(regs, dr6, !dr6);
-	idtentry_exit(regs);
+	idtentry_exit_user(regs);
 }
 
 #ifdef CONFIG_X86_64
#19 lore raw

PATCH v9 7/39 x86/entry: Cleanup idtentry_enter/exit() leftovers

In reply to Thomas Gleixner

Now that everything is converted to conditional RCU handling remove idtentry_enter/exit() and tidy up the conditional functions.

This does not remove rcu_irq_exit_preempt() to avoid conflicts with the RCU tree. Will be removed once all of this hit Linus tree.

---
V9: New patch

---
 arch/x86/entry/common.c         |   67 +++++++++++++++++-----------------------
 arch/x86/include/asm/idtentry.h |   12 -------
 2 files changed, 30 insertions(+), 49 deletions(-)
arch/x86/entry/common.c +29 -38
@@ -515,7 +515,6 @@ SYSCALL_DEFINE0(ni_syscall)
  * idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional
  *			     RCU handling
  * @regs:	Pointer to pt_regs of interrupted context
- * @cond_rcu:	Invoke rcu_irq_enter() only if RCU is not watching
  *
  * Invokes:
  *  - lockdep irqflag state tracking as low level ASM entry disabled
@@ -545,12 +544,12 @@ SYSCALL_DEFINE0(ni_syscall)
  * The return value must be fed into the rcu_exit argument of
  * idtentry_exit_cond_rcu().
  */
-bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu)
+bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs)
 {
 	if (user_mode(regs)) {
 		enter_from_user_mode();
 	} else {
-		if (!cond_rcu || !__rcu_is_watching()) {
+		if (!__rcu_is_watching()) {
 			/*
 			 * If RCU is not watching then the same careful
 			 * sequence vs. lockdep and tracing is required
@@ -605,52 +604,44 @@ void noinstr idtentry_exit_cond_rcu(stru
 	if (user_mode(regs)) {
 		prepare_exit_to_usermode(regs);
 	} else if (regs->flags & X86_EFLAGS_IF) {
+		/*
+		 * If RCU was not watching on entry this needs to be done
+		 * carefully and needs the same ordering of lockdep/tracing
+		 * and RCU as the return to user mode path.
+		 */
+		if (rcu_exit) {
+			instrumentation_begin();
+			/* Tell the tracer that IRET will enable interrupts */
+			trace_hardirqs_on_prepare();
+			lockdep_hardirqs_on_prepare(CALLER_ADDR0);
+			instrumentation_end();
+			rcu_irq_exit();
+			lockdep_hardirqs_on(CALLER_ADDR0);
+			return;
+		}
+
+		instrumentation_begin();
+
 		/* Check kernel preemption, if enabled */
 		if (IS_ENABLED(CONFIG_PREEMPTION)) {
-			/*
-			 * This needs to be done very carefully.
-			 * idtentry_enter() invoked rcu_irq_enter(). This
-			 * needs to be undone before scheduling.
-			 *
-			 * Preemption is disabled inside of RCU idle
-			 * sections. When the task returns from
-			 * preempt_schedule_irq(), RCU is still watching.
-			 *
-			 * rcu_irq_exit_preempt() has additional state
-			 * checking if CONFIG_PROVE_RCU=y
-			 */
 			if (!preempt_count()) {
+				/* Sanity check RCU and thread stack */
+				rcu_irq_exit_check_preempt();
 				if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
 					WARN_ON_ONCE(!on_thread_stack());
-				instrumentation_begin();
-				if (rcu_exit)
-					rcu_irq_exit_preempt();
 				if (need_resched())
 					preempt_schedule_irq();
-				/* Covers both tracing and lockdep */
-				trace_hardirqs_on();
-				instrumentation_end();
-				return;
 			}
 		}
-		/*
-		 * If preemption is disabled then this needs to be done
-		 * carefully with respect to RCU. The exception might come
-		 * from a RCU idle section in the idle task due to the fact
-		 * that safe_halt() enables interrupts. So this needs the
-		 * same ordering of lockdep/tracing and RCU as the return
-		 * to user mode path.
-		 */
-		instrumentation_begin();
-		/* Tell the tracer that IRET will enable interrupts */
-		trace_hardirqs_on_prepare();
-		lockdep_hardirqs_on_prepare(CALLER_ADDR0);
+		/* Covers both tracing and lockdep */
+		trace_hardirqs_on();
+
 		instrumentation_end();
-		if (rcu_exit)
-			rcu_irq_exit();
-		lockdep_hardirqs_on(CALLER_ADDR0);
 	} else {
-		/* IRQ flags state is correct already. Just tell RCU. */
+		/*
+		 * IRQ flags state is correct already. Just tell RCU if it
+		 * was not watching on entry.
+		 */
 		if (rcu_exit)
 			rcu_irq_exit();
 	}
arch/x86/include/asm/idtentry.h +1 -11
@@ -10,19 +10,9 @@
 void idtentry_enter_user(struct pt_regs *regs);
 void idtentry_exit_user(struct pt_regs *regs);
 
-bool idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu);
+bool idtentry_enter_cond_rcu(struct pt_regs *regs);
 void idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit);
 
-static __always_inline void idtentry_enter(struct pt_regs *regs)
-{
-	idtentry_enter_cond_rcu(regs, false);
-}
-
-static __always_inline void idtentry_exit(struct pt_regs *regs)
-{
-	idtentry_exit_cond_rcu(regs, true);
-}
-
 /**
  * DECLARE_IDTENTRY - Declare functions for simple IDT entry points
  *		      No error code pushed by hardware
#20 lore raw

[tip: x86/entry] x86/entry: Clean up idtentry_enter/exit() leftovers

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     1951bc04047a0ca5c88146c4b19b296d40e489ef
Gitweb:        https://git.kernel.org/tip/1951bc04047a0ca5c88146c4b19b296d40e489ef
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:20 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

x86/entry: Clean up idtentry_enter/exit() leftovers

Now that everything is converted to conditional RCU handling remove idtentry_enter/exit() and tidy up the conditional functions.

This does not remove rcu_irq_exit_preempt(), to avoid conflicts with the RCU tree. Will be removed once all of this hits Linus's tree.

---
 arch/x86/entry/common.c         | 67 +++++++++++++-------------------
 arch/x86/include/asm/idtentry.h | 12 +------
 2 files changed, 30 insertions(+), 49 deletions(-)
arch/x86/entry/common.c +29 -38
index b7fcb13..2a80e4e 100644
@@ -515,7 +515,6 @@ SYSCALL_DEFINE0(ni_syscall)
  * idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional
  *			     RCU handling
  * @regs:	Pointer to pt_regs of interrupted context
- * @cond_rcu:	Invoke rcu_irq_enter() only if RCU is not watching
  *
  * Invokes:
  *  - lockdep irqflag state tracking as low level ASM entry disabled
@@ -545,14 +544,14 @@ SYSCALL_DEFINE0(ni_syscall)
  * The return value must be fed into the rcu_exit argument of
  * idtentry_exit_cond_rcu().
  */
-bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu)
+bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs)
 {
 	if (user_mode(regs)) {
 		enter_from_user_mode();
 		return false;
 	}
 
-	if (!cond_rcu || !__rcu_is_watching()) {
+	if (!__rcu_is_watching()) {
 		/*
 		 * If RCU is not watching then the same careful
 		 * sequence vs. lockdep and tracing is required
@@ -608,52 +607,44 @@ void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit)
 	if (user_mode(regs)) {
 		prepare_exit_to_usermode(regs);
 	} else if (regs->flags & X86_EFLAGS_IF) {
+		/*
+		 * If RCU was not watching on entry this needs to be done
+		 * carefully and needs the same ordering of lockdep/tracing
+		 * and RCU as the return to user mode path.
+		 */
+		if (rcu_exit) {
+			instrumentation_begin();
+			/* Tell the tracer that IRET will enable interrupts */
+			trace_hardirqs_on_prepare();
+			lockdep_hardirqs_on_prepare(CALLER_ADDR0);
+			instrumentation_end();
+			rcu_irq_exit();
+			lockdep_hardirqs_on(CALLER_ADDR0);
+			return;
+		}
+
+		instrumentation_begin();
+
 		/* Check kernel preemption, if enabled */
 		if (IS_ENABLED(CONFIG_PREEMPTION)) {
-			/*
-			 * This needs to be done very carefully.
-			 * idtentry_enter() invoked rcu_irq_enter(). This
-			 * needs to be undone before scheduling.
-			 *
-			 * Preemption is disabled inside of RCU idle
-			 * sections. When the task returns from
-			 * preempt_schedule_irq(), RCU is still watching.
-			 *
-			 * rcu_irq_exit_preempt() has additional state
-			 * checking if CONFIG_PROVE_RCU=y
-			 */
 			if (!preempt_count()) {
+				/* Sanity check RCU and thread stack */
+				rcu_irq_exit_check_preempt();
 				if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
 					WARN_ON_ONCE(!on_thread_stack());
-				instrumentation_begin();
-				if (rcu_exit)
-					rcu_irq_exit_preempt();
 				if (need_resched())
 					preempt_schedule_irq();
-				/* Covers both tracing and lockdep */
-				trace_hardirqs_on();
-				instrumentation_end();
-				return;
 			}
 		}
-		/*
-		 * If preemption is disabled then this needs to be done
-		 * carefully with respect to RCU. The exception might come
-		 * from a RCU idle section in the idle task due to the fact
-		 * that safe_halt() enables interrupts. So this needs the
-		 * same ordering of lockdep/tracing and RCU as the return
-		 * to user mode path.
-		 */
-		instrumentation_begin();
-		/* Tell the tracer that IRET will enable interrupts */
-		trace_hardirqs_on_prepare();
-		lockdep_hardirqs_on_prepare(CALLER_ADDR0);
+		/* Covers both tracing and lockdep */
+		trace_hardirqs_on();
+
 		instrumentation_end();
-		if (rcu_exit)
-			rcu_irq_exit();
-		lockdep_hardirqs_on(CALLER_ADDR0);
 	} else {
-		/* IRQ flags state is correct already. Just tell RCU. */
+		/*
+		 * IRQ flags state is correct already. Just tell RCU if it
+		 * was not watching on entry.
+		 */
 		if (rcu_exit)
 			rcu_irq_exit();
 	}
arch/x86/include/asm/idtentry.h +1 -11
index 0f974e5..b056889 100644
@@ -10,19 +10,9 @@
 void idtentry_enter_user(struct pt_regs *regs);
 void idtentry_exit_user(struct pt_regs *regs);
 
-bool idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu);
+bool idtentry_enter_cond_rcu(struct pt_regs *regs);
 void idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit);
 
-static __always_inline void idtentry_enter(struct pt_regs *regs)
-{
-	idtentry_enter_cond_rcu(regs, false);
-}
-
-static __always_inline void idtentry_exit(struct pt_regs *regs)
-{
-	idtentry_exit_cond_rcu(regs, true);
-}
-
 /**
  * DECLARE_IDTENTRY - Declare functions for simple IDT entry points
  *		      No error code pushed by hardware
#21 lore raw

PATCH v9 8/39 genirq: Provide irq_enter/exit_rcu()

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

irq_enter()/exit() include the RCU handling. To properly separate the RCU handling provide variants which contain only the non-RCU related functionality.

---
 include/linux/hardirq.h |   13 +++++++++++--
 kernel/softirq.c        |   35 +++++++++++++++++++++++++++--------
 2 files changed, 38 insertions(+), 10 deletions(-)
include/linux/hardirq.h +11 -2
@@ -52,7 +52,11 @@ static __always_inline void rcu_irq_ente
 /*
  * Enter irq context (on NO_HZ, update jiffies):
  */
-extern void irq_enter(void);
+void irq_enter(void);
+/*
+ * Like irq_enter(), but RCU is already watching.
+ */
+void irq_enter_rcu(void);
 
 /*
  * Exit irq context without processing softirqs:
@@ -67,7 +71,12 @@ extern void irq_enter(void);
 /*
  * Exit irq context and process softirqs if needed:
  */
-extern void irq_exit(void);
+void irq_exit(void);
+
+/*
+ * Like irq_exit(), but return with RCU watching.
+ */
+void irq_exit_rcu(void);
 
 #ifndef arch_nmi_enter
 #define arch_nmi_enter()	do { } while (0)
kernel/softirq.c +27 -8
@@ -339,12 +339,11 @@ asmlinkage __visible void do_softirq(voi
 	local_irq_restore(flags);
 }
 
-/*
- * Enter an interrupt context.
+/**
+ * irq_enter_rcu - Enter an interrupt context with RCU watching
  */
-void irq_enter(void)
+void irq_enter_rcu(void)
 {
-	rcu_irq_enter();
 	if (is_idle_task(current) && !in_interrupt()) {
 		/*
 		 * Prevent raise_softirq from needlessly waking up ksoftirqd
@@ -354,10 +353,18 @@ void irq_enter(void)
 		tick_irq_enter();
 		_local_bh_enable();
 	}
-
 	__irq_enter();
 }
 
+/**
+ * irq_enter - Enter an interrupt context including RCU update
+ */
+void irq_enter(void)
+{
+	rcu_irq_enter();
+	irq_enter_rcu();
+}
+
 static inline void invoke_softirq(void)
 {
 	if (ksoftirqd_running(local_softirq_pending()))
@@ -397,10 +404,12 @@ static inline void tick_irq_exit(void)
 #endif
 }
 
-/*
- * Exit an interrupt context. Process softirqs if needed and possible:
+/**
+ * irq_exit_rcu() - Exit an interrupt context without updating RCU
+ *
+ * Also processes softirqs if needed and possible.
  */
-void irq_exit(void)
+void irq_exit_rcu(void)
 {
 #ifndef __ARCH_IRQ_EXIT_IRQS_DISABLED
 	local_irq_disable();
@@ -413,6 +422,16 @@ void irq_exit(void)
 		invoke_softirq();
 
 	tick_irq_exit();
+}
+
+/**
+ * irq_exit - Exit an interrupt context, update RCU and lockdep
+ *
+ * Also processes softirqs if needed and possible.
+ */
+void irq_exit(void)
+{
+	irq_exit_rcu();
 	rcu_irq_exit();
 	 /* must be last! */
 	lockdep_hardirq_exit();
#22 lore raw

[tip: x86/entry] genirq: Provide irq_enter/exit_rcu()

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     a59262f6413b82992626754b46383a615bfac152
Gitweb:        https://git.kernel.org/tip/a59262f6413b82992626754b46383a615bfac152
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:21 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

genirq: Provide irq_enter/exit_rcu()

irq_enter()/exit() currently include RCU handling. To properly separate the RCU handling code, provide variants which contain only the non-RCU related functionality.

---
 include/linux/hardirq.h | 13 +++++++++++--
 kernel/softirq.c        | 35 +++++++++++++++++++++++++++--------
 2 files changed, 38 insertions(+), 10 deletions(-)
include/linux/hardirq.h +11 -2
index 29b862a..3dc9102 100644
@@ -40,7 +40,11 @@ static __always_inline void rcu_irq_enter_check_tick(void)
 /*
  * Enter irq context (on NO_HZ, update jiffies):
  */
-extern void irq_enter(void);
+void irq_enter(void);
+/*
+ * Like irq_enter(), but RCU is already watching.
+ */
+void irq_enter_rcu(void);
 
 /*
  * Exit irq context without processing softirqs:
@@ -55,7 +59,12 @@ extern void irq_enter(void);
 /*
  * Exit irq context and process softirqs if needed:
  */
-extern void irq_exit(void);
+void irq_exit(void);
+
+/*
+ * Like irq_exit(), but return with RCU watching.
+ */
+void irq_exit_rcu(void);
 
 #ifndef arch_nmi_enter
 #define arch_nmi_enter()	do { } while (0)
kernel/softirq.c +27 -8
index a47c6dd..beb8e3a 100644
@@ -339,12 +339,11 @@ asmlinkage __visible void do_softirq(void)
 	local_irq_restore(flags);
 }
 
-/*
- * Enter an interrupt context.
+/**
+ * irq_enter_rcu - Enter an interrupt context with RCU watching
  */
-void irq_enter(void)
+void irq_enter_rcu(void)
 {
-	rcu_irq_enter();
 	if (is_idle_task(current) && !in_interrupt()) {
 		/*
 		 * Prevent raise_softirq from needlessly waking up ksoftirqd
@@ -354,10 +353,18 @@ void irq_enter(void)
 		tick_irq_enter();
 		_local_bh_enable();
 	}
-
 	__irq_enter();
 }
 
+/**
+ * irq_enter - Enter an interrupt context including RCU update
+ */
+void irq_enter(void)
+{
+	rcu_irq_enter();
+	irq_enter_rcu();
+}
+
 static inline void invoke_softirq(void)
 {
 	if (ksoftirqd_running(local_softirq_pending()))
@@ -397,10 +404,12 @@ static inline void tick_irq_exit(void)
 #endif
 }
 
-/*
- * Exit an interrupt context. Process softirqs if needed and possible:
+/**
+ * irq_exit_rcu() - Exit an interrupt context without updating RCU
+ *
+ * Also processes softirqs if needed and possible.
  */
-void irq_exit(void)
+void irq_exit_rcu(void)
 {
 #ifndef __ARCH_IRQ_EXIT_IRQS_DISABLED
 	local_irq_disable();
@@ -413,6 +422,16 @@ void irq_exit(void)
 		invoke_softirq();
 
 	tick_irq_exit();
+}
+
+/**
+ * irq_exit - Exit an interrupt context, update RCU and lockdep
+ *
+ * Also processes softirqs if needed and possible.
+ */
+void irq_exit(void)
+{
+	irq_exit_rcu();
 	rcu_irq_exit();
 	 /* must be last! */
 	lockdep_hardirq_exit();
#23 lore raw

PATCH v9 9/39 genirq: Provide __irq_enter/exit_raw()

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Like __irq_enter/exit() but without time accounting. To be used for "empty" system vectors like the scheduler IPI to avoid the overhead.

---
 include/linux/hardirq.h |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
include/linux/hardirq.h +20 -0
@@ -50,6 +50,17 @@ static __always_inline void rcu_irq_ente
 	} while (0)
 
 /*
+ * Like __irq_enter() without time accounting for fast
+ * interrupts, e.g. reschedule IPI where time accounting
+ * is more expensive than the actual interrupt.
+ */
+#define __irq_enter_raw()				\
+	do {						\
+		preempt_count_add(HARDIRQ_OFFSET);	\
+		lockdep_hardirq_enter();		\
+	} while (0)
+
+/*
  * Enter irq context (on NO_HZ, update jiffies):
  */
 void irq_enter(void);
@@ -68,6 +79,15 @@ void irq_enter_rcu(void);
 		preempt_count_sub(HARDIRQ_OFFSET);	\
 	} while (0)
 
+/*
+ * Like __irq_exit() without time accounting
+ */
+#define __irq_exit_raw()				\
+	do {						\
+		lockdep_hardirq_exit();			\
+		preempt_count_sub(HARDIRQ_OFFSET);	\
+	} while (0)
+
 /*
  * Exit irq context and process softirqs if needed:
  */
#24 lore raw

[tip: x86/entry] genirq: Provide __irq_enter/exit_raw()

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     355e1262d6038b37f0dbf5f3cfe64cc85d471b7b
Gitweb:        https://git.kernel.org/tip/355e1262d6038b37f0dbf5f3cfe64cc85d471b7b
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:22 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

genirq: Provide __irq_enter/exit_raw()

Like __irq_enter/exit() but without time accounting. To be used for "empty" system vectors like the scheduler IPI to avoid the overhead.

---
 include/linux/hardirq.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
include/linux/hardirq.h +20 -0
index 3dc9102..03c9fec 100644
@@ -38,6 +38,17 @@ static __always_inline void rcu_irq_enter_check_tick(void)
 	} while (0)
 
 /*
+ * Like __irq_enter() without time accounting for fast
+ * interrupts, e.g. reschedule IPI where time accounting
+ * is more expensive than the actual interrupt.
+ */
+#define __irq_enter_raw()				\
+	do {						\
+		preempt_count_add(HARDIRQ_OFFSET);	\
+		lockdep_hardirq_enter();		\
+	} while (0)
+
+/*
  * Enter irq context (on NO_HZ, update jiffies):
  */
 void irq_enter(void);
@@ -57,6 +68,15 @@ void irq_enter_rcu(void);
 	} while (0)
 
 /*
+ * Like __irq_exit() without time accounting
+ */
+#define __irq_exit_raw()				\
+	do {						\
+		lockdep_hardirq_exit();			\
+		preempt_count_sub(HARDIRQ_OFFSET);	\
+	} while (0)
+
+/*
  * Exit irq context and process softirqs if needed:
  */
 void irq_exit(void);
#25 lore raw

PATCH v9 10/39 x86/entry: Provide helpers for execute on irqstack

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.

As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.

That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.

The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.

Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.

64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.

---
V9: Moved the conditions into an inline to avoid code duplication

---
 arch/x86/entry/entry_64.S        |   39 ++++++++++++++++++++++++++++
 arch/x86/include/asm/irq_stack.h |   53 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+)
arch/x86/entry/entry_64.S +39 -0
@@ -1106,6 +1106,45 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs)
 SYM_CODE_END(.Lbad_gs)
 	.previous
 
+/*
+ * rdi: New stack pointer points to the top word of the stack
+ * rsi: Function pointer
+ * rdx: Function argument (can be NULL if none)
+ */
+SYM_FUNC_START(asm_call_on_stack)
+	/*
+	 * Save the frame pointer unconditionally. This allows the ORC
+	 * unwinder to handle the stack switch.
+	 */
+	pushq		%rbp
+	mov		%rsp, %rbp
+
+	/*
+	 * The unwinder relies on the word at the top of the new stack
+	 * page linking back to the previous RSP.
+	 */
+	mov		%rsp, (%rdi)
+	mov		%rdi, %rsp
+	/* Move the argument to the right place */
+	mov		%rdx, %rdi
+
+1:
+	.pushsection .discard.instr_begin
+	.long 1b - .
+	.popsection
+
+	CALL_NOSPEC	rsi
+
+2:
+	.pushsection .discard.instr_end
+	.long 2b - .
+	.popsection
+
+	/* Restore the previous stack pointer from RBP. */
+	leaveq
+	ret
+SYM_FUNC_END(asm_call_on_stack)
+
 /* Call softirq on interrupt stack. Interrupts are off. */
 .pushsection .text, "ax"
 SYM_FUNC_START(do_softirq_own_stack)
arch/x86/include/asm/irq_stack.h new file +53 -0

53 lines not on this page · raw message

#26 lore raw

[tip: x86/entry] x86/entry: Provide helpers for executing on the irqstack

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     0aa4dbb2808991f53396df8d2deb390d4f880abb
Gitweb:        https://git.kernel.org/tip/0aa4dbb2808991f53396df8d2deb390d4f880abb
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:23 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

x86/entry: Provide helpers for executing on the irqstack

Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.

As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.

That allows the moving of the stack switching to C code, which simplifies the entry/exit handling further, because it allows the switching of stacks after handling the entry and on exit before handling RCU, returning to usermode and kernel preemption in the same way as for regular exceptions.

The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable, as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.

Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.

64-bit only for now, as 32-bit has a variant of that already. Once this is cleaned up, the two implementations might be consolidated as an additional cleanup on top.

---
 arch/x86/entry/entry_64.S        | 39 +++++++++++++++++++++++-
 arch/x86/include/asm/irq_stack.h | 53 +++++++++++++++++++++++++++++++-
 2 files changed, 92 insertions(+)
 create mode 100644 arch/x86/include/asm/irq_stack.h
arch/x86/entry/entry_64.S +39 -0
index d983a0d..1597370 100644
@@ -1106,6 +1106,45 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs)
 SYM_CODE_END(.Lbad_gs)
 	.previous
 
+/*
+ * rdi: New stack pointer points to the top word of the stack
+ * rsi: Function pointer
+ * rdx: Function argument (can be NULL if none)
+ */
+SYM_FUNC_START(asm_call_on_stack)
+	/*
+	 * Save the frame pointer unconditionally. This allows the ORC
+	 * unwinder to handle the stack switch.
+	 */
+	pushq		%rbp
+	mov		%rsp, %rbp
+
+	/*
+	 * The unwinder relies on the word at the top of the new stack
+	 * page linking back to the previous RSP.
+	 */
+	mov		%rsp, (%rdi)
+	mov		%rdi, %rsp
+	/* Move the argument to the right place */
+	mov		%rdx, %rdi
+
+1:
+	.pushsection .discard.instr_begin
+	.long 1b - .
+	.popsection
+
+	CALL_NOSPEC	rsi
+
+2:
+	.pushsection .discard.instr_end
+	.long 2b - .
+	.popsection
+
+	/* Restore the previous stack pointer from RBP. */
+	leaveq
+	ret
+SYM_FUNC_END(asm_call_on_stack)
+
 /* Call softirq on interrupt stack. Interrupts are off. */
 .pushsection .text, "ax"
 SYM_FUNC_START(do_softirq_own_stack)
arch/x86/include/asm/irq_stack.h new file +53 -0

53 lines not on this page · raw message

#27 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

In reply to Thomas Gleixner
Quoted 23 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.

As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.

That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.

The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.

Quoted 8 lines

Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.

64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.

Reverted this commit and the rest of series (with trivial fixup) as well as the two dependencies [1],

8449e768dcb8 ("x86/entry: Remove debug IDT frobbing") 029149180d1d ("x86/entry: Rename trace_hardirqs_off_prepare()")

fixed the warning under some memory pressure on AMD NUMA servers.

[ 9371.959858]  asm_call_on_stack+0x12/0x20

asm_call_on_stack at arch/x86/entry/entry_64.S:710

The .config (if ever matters),
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config

[ 9371.260161] ------------[ cut here ]------------
[ 9371.267143] Stack depot reached limit capacity
[ 9371.267193] WARNING: CPU: 19 PID: 1181 at lib/stackdepot.c:115 stack_depot_save+0x3d9/0x57d
[ 9371.281470] Modules linked in: brd vfat fat ext4 crc16 mbcache jbd2 loop kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core acpi_cpufreq ip_tables x_table
s xfs sd_mod bnxt_en smartpqi scsi_transport_sas tg3 i40e libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_del_mod]
[ 9371.310176] CPU: 19 PID: 1181 Comm: systemd-journal Tainted: G           O      5.7.0-next-20200604+ #1
[ 9371.320700] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 03/09/2018
[ 9371.329987] RIP: 0010:stack_depot_save+0x3d9/0x57d
[ 9371.335513] Code: 1d 9b bc 68 01 80 fb 01 0f 87 c0 01 00 00 80 e3 01 75 1f 4c 89 45 c0 c6 05 82 bc 68 01 01 48 c7 c7 e0 85 63 9f e8 9d 74 9d ff <0f> 0b 90 90 4c 8
b 45 c0 48 c7 c7 80 1a d0 9f 4c 89 c6 e8 b0 2b 46
[ 9371.355426] RSP: 0018:ffffc90007260490 EFLAGS: 00010082
[ 9371.361387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff9ed3207f
[ 9371.369544] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: 0000000000000000
[ 9371.377428] RBP: ffffc900072604f8 R08: fffffbfff3f37539 R09: fffffbfff3f37539
[ 9371.385310] R10: ffffffff9f9ba9c3 R11: fffffbfff3f37538 R12: ffffc90007260508
[ 9371.393521] R13: 0000000000000036 R14: 0000000000000000 R15: 000000000009fb52
[ 9371.401403] FS:  00007fc9849f2980(0000) GS:ffff88942fb80000(0000) knlGS:0000000000000000
[ 9371.410244] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9371.417007] CR2: 00007f20d02c3000 CR3: 0000000440b9c000 CR4: 00000000003406e0
[ 9371.424889] Call Trace:
[ 9371.428054]  <IRQ>
[ 9371.436315]  save_stack+0x3f/0x50
[ 9371.734034]  kasan_slab_free+0xe/0x10
[ 9371.738798]  slab_free_freelist_hook+0x5d/0x1c0
[ 9371.748886]  kmem_cache_free+0x10c/0x390
[ 9371.758450]  mempool_free_slab+0x17/0x20
[ 9371.763522]  mempool_free+0x65/0x170
[ 9371.767825]  bio_free+0x14c/0x210
[ 9371.771864]  bio_put+0x59/0x70
[ 9371.775644]  end_swap_bio_write+0x199/0x250
[ 9371.780556]  bio_endio+0x22c/0x4e0
[ 9371.784709]  dec_pending+0x1bf/0x3e0 [dm_mod]
[ 9371.790207]  clone_endio+0x129/0x3d0 [dm_mod]
[ 9371.800746]  bio_endio+0x22c/0x4e0
[ 9371.809262]  blk_update_request+0x3bb/0x980
[ 9371.814605]  scsi_end_request+0x53/0x420
[ 9371.824002]  scsi_io_completion+0x10a/0x830
[ 9371.844445]  scsi_finish_command+0x1b9/0x250
[ 9371.849445]  scsi_softirq_done+0x1ab/0x1f0
[ 9371.854272]  blk_mq_force_complete_rq+0x217/0x250
[ 9371.859708]  blk_mq_complete_request+0xe/0x20
[ 9371.865171]  scsi_mq_done+0xc1/0x220
[ 9371.869479]  pqi_aio_io_complete+0x83/0x2c0 [smartpqi]
[ 9371.881764]  pqi_irq_handler+0x1fc/0x13f0 [smartpqi]
[ 9371.914115]  __handle_irq_event_percpu+0x81/0x550
[ 9371.924289]  handle_irq_event_percpu+0x70/0x100
[ 9371.945559]  handle_irq_event+0x5a/0x8b
[ 9371.950121]  handle_edge_irq+0x10c/0x370
[ 9371.950121]  handle_edge_irq+0x10c/0x370
[ 9371.959858]  asm_call_on_stack+0x12/0x20
asm_call_on_stack at arch/x86/entry/entry_64.S:710
[ 9371.964899]  </IRQ>
[ 9371.967716]  common_interrupt+0x185/0x2a0
[ 9371.972455]  asm_common_interrupt+0x1e/0x40
[ 9371.977368] RIP: 0010:__asan_load4+0x8/0xa0
[ 9371.982281] Code: 00 e8 5c f4 ff ff 5d c3 40 38 f0 0f 9e c0 84 c0 75 e5 5d c3 48 c1 e8 03 80 3c 10 00 75 ed 5d c3 66 90 55 48 89 e5 48 8b 4d 08 <48> 83 ff fb 77 6c eb 3a 0f 1f 00 48 b8 00 00 00 00 00 00 00 ff 48
[ 9372.002246] RSP: 0018:ffffc9000b12f0e8 EFLAGS: 00000202
[ 9372.008207] RAX: 0000000000000000 RBX: ffffc9000b12f150 RCX: ffffffff9ed32487
[ 9372.016489] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffffffff9ff92a94
[ 9372.024370] RBP: ffffc9000b12f0e8 R08: fffffbfff3ff1d4d R09: fffffbfff3ff1d4d
[ 9372.032252] R10: ffffffff9ff8ea67 R11: fffffbfff3ff1d4c R12: 0000000000000000
[ 9372.040490] R13: ffffc9000b12f358 R14: 000000000000000c R15: 0000000000000005
[ 9372.053899]  debug_lockdep_rcu_enabled+0x27/0x60
[ 9372.059248]  rcu_read_lock_held_common+0x12/0x60
[ 9372.064989]  rcu_read_lock_sched_held+0x60/0xe0
[ 9372.080338]  shrink_active_list+0xbfd/0xc30
[ 9372.120481]  shrink_lruvec+0xbf1/0x11b0
[ 9372.150410]  shrink_node+0x344/0xd10
[ 9372.154719]  do_try_to_free_pages+0x263/0xa00
[ 9372.169860]  try_to_free_pages+0x239/0x570
[ 9372.179356]  __alloc_pages_slowpath.constprop.59+0x5dd/0x1880
[ 9372.215762]  __alloc_pages_nodemask+0x562/0x670
[ 9372.232686]  alloc_pages_current+0x9c/0x110
[ 9372.237599]  alloc_slab_page+0x355/0x530
[ 9372.242755]  allocate_slab+0x485/0x5a0
[ 9372.247232]  new_slab+0x46/0x70
[ 9372.251095]  ___slab_alloc+0x35f/0x810
[ 9372.274485]  __slab_alloc+0x43/0x70
[ 9372.287650]  kmem_cache_alloc+0x257/0x3d0
[ 9372.298182]  prepare_creds+0x26/0x130
[ 9372.302571]  do_faccessat+0x255/0x3e0
[ 9372.321591]  __x64_sys_access+0x38/0x40
[ 9372.326154]  do_syscall_64+0x64/0x340
[ 9372.330542]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9372.336329] RIP: 0033:0x7fc983a21bfb
[ 9372.341050] Code: Bad RIP value.
[ 9372.345000] RSP: 002b:00007fffd543cde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000015
[ 9372.353319] RAX: ffffffffffffffda RBX: 00007fffd543fa40 RCX: 00007fc983a21bfb
[ 9372.361199] RDX: 00007fc983cf1c00 RSI: 0000000000000000 RDI: 00005573ed458090
[ 9372.369512] RBP: 00007fffd543cf30 R08: 00005573ed44ab99 R09: 0000000000000007
[ 9372.377393] R10: 0000000000000041 R11: 0000000000000246 R12: 0000000000000000
[ 9372.385275] R13: 0000000000000000 R14: 00007fffd543cea0 R15: 00005573eecd2990
[ 9372.393608] irq event stamp: 27274496
[ 9372.397999] hardirqs last  enabled at (27274495): [<ffffffff9e635b8e>] free_unref_page_list+0x2ee/0x400
[ 9372.408152] hardirqs last disabled at (27274496): [<ffffffff9ed2c22b>] idtentry_enter_cond_rcu+0x1b/0x50
[ 9372.418695] softirqs last  enabled at (27272816): [<ffffffff9f000478>] __do_softirq+0x478/0x784
[ 9372.428154] softirqs last disabled at (27272807): [<ffffffff9e2d0b41>] irq_exit_rcu+0xd1/0xe0
[ 9372.437435] ---[ end trace d2ebac1fad6e452e ]---

[1]
git revert --no-edit 355e1262d603..5a7462b1f9c1

Quoted 112 lines

---
V9: Moved the conditions into an inline to avoid code duplication

---
 arch/x86/entry/entry_64.S        |   39 ++++++++++++++++++++++++++++
 arch/x86/include/asm/irq_stack.h |   53 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 92 insertions(+)
arch/x86/entry/entry_64.S +39 -0
@@ -1106,6 +1106,45 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs)
 SYM_CODE_END(.Lbad_gs)
 	.previous
 
+/*
+ * rdi: New stack pointer points to the top word of the stack
+ * rsi: Function pointer
+ * rdx: Function argument (can be NULL if none)
+ */
+SYM_FUNC_START(asm_call_on_stack)
+	/*
+	 * Save the frame pointer unconditionally. This allows the ORC
+	 * unwinder to handle the stack switch.
+	 */
+	pushq		%rbp
+	mov		%rsp, %rbp
+
+	/*
+	 * The unwinder relies on the word at the top of the new stack
+	 * page linking back to the previous RSP.
+	 */
+	mov		%rsp, (%rdi)
+	mov		%rdi, %rsp
+	/* Move the argument to the right place */
+	mov		%rdx, %rdi
+
+1:
+	.pushsection .discard.instr_begin
+	.long 1b - .
+	.popsection
+
+	CALL_NOSPEC	rsi
+
+2:
+	.pushsection .discard.instr_end
+	.long 2b - .
+	.popsection
+
+	/* Restore the previous stack pointer from RBP. */
+	leaveq
+	ret
+SYM_FUNC_END(asm_call_on_stack)
+
 /* Call softirq on interrupt stack. Interrupts are off. */
 .pushsection .text, "ax"
 SYM_FUNC_START(do_softirq_own_stack)
arch/x86/include/asm/irq_stack.h new file +53 -0
@@ -0,0 +1,53 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_IRQ_STACK_H
+#define _ASM_X86_IRQ_STACK_H
+
+#include <linux/ptrace.h>
+
+#include <asm/processor.h>
+
+#ifdef CONFIG_X86_64
+static __always_inline bool irqstack_active(void)
+{
+	return __this_cpu_read(irq_count) != -1;
+}
+
+void asm_call_on_stack(void *sp, void *func, void *arg);
+
+static __always_inline void __run_on_irqstack(void *func, void *arg)
+{
+	void *tos = __this_cpu_read(hardirq_stack_ptr);
+
+	__this_cpu_add(irq_count, 1);
+	asm_call_on_stack(tos - 8, func, arg);
+	__this_cpu_sub(irq_count, 1);
+}
+
+#else /* CONFIG_X86_64 */
+static inline bool irqstack_active(void) { return false; }
+static inline void __run_on_irqstack(void *func, void *arg) { }
+#endif /* !CONFIG_X86_64 */
+
+static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs)
+{
+	if (IS_ENABLED(CONFIG_X86_32))
+		return false;
+	if (!regs)
+		return !irqstack_active();
+	return !user_mode(regs) && !irqstack_active();
+}
+
+static __always_inline void run_on_irqstack_cond(void *func, void *arg,
+						 struct pt_regs *regs)
+{
+	void (*__func)(void *arg) = func;
+
+	lockdep_assert_irqs_disabled();
+
+	if (irq_needs_irq_stack(regs))
+		__run_on_irqstack(__func, arg);
+	else
+		__func(arg);
+}
+
+#endif
maintainer
#28 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 33 lines · Qian Cai
Quoted 23 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.

As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.

That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.

The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.

Quoted 8 lines

Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.

64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.

Quoted 7 lines

Reverted this commit and the rest of series (with trivial fixup) as well as the two dependencies [1],

8449e768dcb8 ("x86/entry: Remove debug IDT frobbing") 029149180d1d ("x86/entry: Rename trace_hardirqs_off_prepare()")

fixed the warning under some memory pressure on AMD NUMA servers.

The warning ?

Quoted 2 lines
[ 9371.959858]  asm_call_on_stack+0x12/0x20

asm_call_on_stack at arch/x86/entry/entry_64.S:710

^^ what's that?

Quoted 6 lines

The .config (if ever matters),
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config

[ 9371.260161] ------------[ cut here ]------------
[ 9371.267143] Stack depot reached limit capacity
[ 9371.267193] WARNING: CPU: 19 PID: 1181 at lib/stackdepot.c:115 stack_depot_save+0x3d9/0x57d

Is this _the_ warning?

Maybe it really is running out of storage, I don't think this patch is to blame. The unwind here looks good, so why would the stack-depot unwind be any different?

Quoted 95 lines
[ 9371.281470] Modules linked in: brd vfat fat ext4 crc16 mbcache jbd2 loop kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core acpi_cpufreq ip_tables x_table
s xfs sd_mod bnxt_en smartpqi scsi_transport_sas tg3 i40e libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_del_mod]
[ 9371.310176] CPU: 19 PID: 1181 Comm: systemd-journal Tainted: G           O      5.7.0-next-20200604+ #1
[ 9371.320700] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 03/09/2018
[ 9371.329987] RIP: 0010:stack_depot_save+0x3d9/0x57d
[ 9371.335513] Code: 1d 9b bc 68 01 80 fb 01 0f 87 c0 01 00 00 80 e3 01 75 1f 4c 89 45 c0 c6 05 82 bc 68 01 01 48 c7 c7 e0 85 63 9f e8 9d 74 9d ff <0f> 0b 90 90 4c 8
b 45 c0 48 c7 c7 80 1a d0 9f 4c 89 c6 e8 b0 2b 46
[ 9371.355426] RSP: 0018:ffffc90007260490 EFLAGS: 00010082
[ 9371.361387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff9ed3207f
[ 9371.369544] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: 0000000000000000
[ 9371.377428] RBP: ffffc900072604f8 R08: fffffbfff3f37539 R09: fffffbfff3f37539
[ 9371.385310] R10: ffffffff9f9ba9c3 R11: fffffbfff3f37538 R12: ffffc90007260508
[ 9371.393521] R13: 0000000000000036 R14: 0000000000000000 R15: 000000000009fb52
[ 9371.401403] FS:  00007fc9849f2980(0000) GS:ffff88942fb80000(0000) knlGS:0000000000000000
[ 9371.410244] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9371.417007] CR2: 00007f20d02c3000 CR3: 0000000440b9c000 CR4: 00000000003406e0
[ 9371.424889] Call Trace:
[ 9371.428054]  <IRQ>
[ 9371.436315]  save_stack+0x3f/0x50
[ 9371.734034]  kasan_slab_free+0xe/0x10
[ 9371.738798]  slab_free_freelist_hook+0x5d/0x1c0
[ 9371.748886]  kmem_cache_free+0x10c/0x390
[ 9371.758450]  mempool_free_slab+0x17/0x20
[ 9371.763522]  mempool_free+0x65/0x170
[ 9371.767825]  bio_free+0x14c/0x210
[ 9371.771864]  bio_put+0x59/0x70
[ 9371.775644]  end_swap_bio_write+0x199/0x250
[ 9371.780556]  bio_endio+0x22c/0x4e0
[ 9371.784709]  dec_pending+0x1bf/0x3e0 [dm_mod]
[ 9371.790207]  clone_endio+0x129/0x3d0 [dm_mod]
[ 9371.800746]  bio_endio+0x22c/0x4e0
[ 9371.809262]  blk_update_request+0x3bb/0x980
[ 9371.814605]  scsi_end_request+0x53/0x420
[ 9371.824002]  scsi_io_completion+0x10a/0x830
[ 9371.844445]  scsi_finish_command+0x1b9/0x250
[ 9371.849445]  scsi_softirq_done+0x1ab/0x1f0
[ 9371.854272]  blk_mq_force_complete_rq+0x217/0x250
[ 9371.859708]  blk_mq_complete_request+0xe/0x20
[ 9371.865171]  scsi_mq_done+0xc1/0x220
[ 9371.869479]  pqi_aio_io_complete+0x83/0x2c0 [smartpqi]
[ 9371.881764]  pqi_irq_handler+0x1fc/0x13f0 [smartpqi]
[ 9371.914115]  __handle_irq_event_percpu+0x81/0x550
[ 9371.924289]  handle_irq_event_percpu+0x70/0x100
[ 9371.945559]  handle_irq_event+0x5a/0x8b
[ 9371.950121]  handle_edge_irq+0x10c/0x370
[ 9371.950121]  handle_edge_irq+0x10c/0x370
[ 9371.959858]  asm_call_on_stack+0x12/0x20
asm_call_on_stack at arch/x86/entry/entry_64.S:710
[ 9371.964899]  </IRQ>
[ 9371.967716]  common_interrupt+0x185/0x2a0
[ 9371.972455]  asm_common_interrupt+0x1e/0x40
[ 9371.977368] RIP: 0010:__asan_load4+0x8/0xa0
[ 9371.982281] Code: 00 e8 5c f4 ff ff 5d c3 40 38 f0 0f 9e c0 84 c0 75 e5 5d c3 48 c1 e8 03 80 3c 10 00 75 ed 5d c3 66 90 55 48 89 e5 48 8b 4d 08 <48> 83 ff fb 77 6c eb 3a 0f 1f 00 48 b8 00 00 00 00 00 00 00 ff 48
[ 9372.002246] RSP: 0018:ffffc9000b12f0e8 EFLAGS: 00000202
[ 9372.008207] RAX: 0000000000000000 RBX: ffffc9000b12f150 RCX: ffffffff9ed32487
[ 9372.016489] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffffffff9ff92a94
[ 9372.024370] RBP: ffffc9000b12f0e8 R08: fffffbfff3ff1d4d R09: fffffbfff3ff1d4d
[ 9372.032252] R10: ffffffff9ff8ea67 R11: fffffbfff3ff1d4c R12: 0000000000000000
[ 9372.040490] R13: ffffc9000b12f358 R14: 000000000000000c R15: 0000000000000005
[ 9372.053899]  debug_lockdep_rcu_enabled+0x27/0x60
[ 9372.059248]  rcu_read_lock_held_common+0x12/0x60
[ 9372.064989]  rcu_read_lock_sched_held+0x60/0xe0
[ 9372.080338]  shrink_active_list+0xbfd/0xc30
[ 9372.120481]  shrink_lruvec+0xbf1/0x11b0
[ 9372.150410]  shrink_node+0x344/0xd10
[ 9372.154719]  do_try_to_free_pages+0x263/0xa00
[ 9372.169860]  try_to_free_pages+0x239/0x570
[ 9372.179356]  __alloc_pages_slowpath.constprop.59+0x5dd/0x1880
[ 9372.215762]  __alloc_pages_nodemask+0x562/0x670
[ 9372.232686]  alloc_pages_current+0x9c/0x110
[ 9372.237599]  alloc_slab_page+0x355/0x530
[ 9372.242755]  allocate_slab+0x485/0x5a0
[ 9372.247232]  new_slab+0x46/0x70
[ 9372.251095]  ___slab_alloc+0x35f/0x810
[ 9372.274485]  __slab_alloc+0x43/0x70
[ 9372.287650]  kmem_cache_alloc+0x257/0x3d0
[ 9372.298182]  prepare_creds+0x26/0x130
[ 9372.302571]  do_faccessat+0x255/0x3e0
[ 9372.321591]  __x64_sys_access+0x38/0x40
[ 9372.326154]  do_syscall_64+0x64/0x340
[ 9372.330542]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9372.336329] RIP: 0033:0x7fc983a21bfb
[ 9372.341050] Code: Bad RIP value.
[ 9372.345000] RSP: 002b:00007fffd543cde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000015
[ 9372.353319] RAX: ffffffffffffffda RBX: 00007fffd543fa40 RCX: 00007fc983a21bfb
[ 9372.361199] RDX: 00007fc983cf1c00 RSI: 0000000000000000 RDI: 00005573ed458090
[ 9372.369512] RBP: 00007fffd543cf30 R08: 00005573ed44ab99 R09: 0000000000000007
[ 9372.377393] R10: 0000000000000041 R11: 0000000000000246 R12: 0000000000000000
[ 9372.385275] R13: 0000000000000000 R14: 00007fffd543cea0 R15: 00005573eecd2990
[ 9372.393608] irq event stamp: 27274496
[ 9372.397999] hardirqs last  enabled at (27274495): [<ffffffff9e635b8e>] free_unref_page_list+0x2ee/0x400
[ 9372.408152] hardirqs last disabled at (27274496): [<ffffffff9ed2c22b>] idtentry_enter_cond_rcu+0x1b/0x50
[ 9372.418695] softirqs last  enabled at (27272816): [<ffffffff9f000478>] __do_softirq+0x478/0x784
[ 9372.428154] softirqs last disabled at (27272807): [<ffffffff9e2d0b41>] irq_exit_rcu+0xd1/0xe0
[ 9372.437435] ---[ end trace d2ebac1fad6e452e ]---
#29 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 37 lines · Peter Zijlstra
Quoted 36 lines · Qian Cai
Quoted 23 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.

As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.

That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.

The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.

Quoted 8 lines

Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.

64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.

Reverted this commit and the rest of series (with trivial fixup) as well as the two dependencies [1],

Quoted 6 lines
Quoted 4 lines

8449e768dcb8 ("x86/entry: Remove debug IDT frobbing") 029149180d1d ("x86/entry: Rename trace_hardirqs_off_prepare()")

fixed the warning under some memory pressure on AMD NUMA servers.

The warning ?

Obviously, the warning below.

Quoted 3 lines
Quoted 3 lines
[ 9371.959858]  asm_call_on_stack+0x12/0x20

asm_call_on_stack at arch/x86/entry/entry_64.S:710

This is one piece of call from the warning call traces that introduced by the patch which leads me to revert the commit in the first place. It may or may not be the real culprit, but just wanted to highlight it in case.

Quoted 10 lines

^^ what's that?

Quoted 6 lines

The .config (if ever matters),
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config

[ 9371.260161] ------------[ cut here ]------------
[ 9371.267143] Stack depot reached limit capacity
[ 9371.267193] WARNING: CPU: 19 PID: 1181 at lib/stackdepot.c:115 stack_depot_save+0x3d9/0x57d
Quoted 5 lines

Is this _the_ warning?

Maybe it really is running out of storage, I don't think this patch is to blame. The unwind here looks good, so why would the stack-depot unwind be any different?

Don't know. It is always reproducible with the same warning and call traces on multiple similar AMD servers. Can this patch (or series) increase the storage so much or some kind of recursion? Otherwise, never saw this problem before the series was introduced. It is always running almost the same tests for years on those machines.

Quoted 96 lines
Quoted 96 lines
[ 9371.281470] Modules linked in: brd vfat fat ext4 crc16 mbcache jbd2 loop kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core acpi_cpufreq ip_tables x_table
s xfs sd_mod bnxt_en smartpqi scsi_transport_sas tg3 i40e libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_del_mod]
[ 9371.310176] CPU: 19 PID: 1181 Comm: systemd-journal Tainted: G           O      5.7.0-next-20200604+ #1
[ 9371.320700] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 03/09/2018
[ 9371.329987] RIP: 0010:stack_depot_save+0x3d9/0x57d
[ 9371.335513] Code: 1d 9b bc 68 01 80 fb 01 0f 87 c0 01 00 00 80 e3 01 75 1f 4c 89 45 c0 c6 05 82 bc 68 01 01 48 c7 c7 e0 85 63 9f e8 9d 74 9d ff <0f> 0b 90 90 4c 8
b 45 c0 48 c7 c7 80 1a d0 9f 4c 89 c6 e8 b0 2b 46
[ 9371.355426] RSP: 0018:ffffc90007260490 EFLAGS: 00010082
[ 9371.361387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff9ed3207f
[ 9371.369544] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: 0000000000000000
[ 9371.377428] RBP: ffffc900072604f8 R08: fffffbfff3f37539 R09: fffffbfff3f37539
[ 9371.385310] R10: ffffffff9f9ba9c3 R11: fffffbfff3f37538 R12: ffffc90007260508
[ 9371.393521] R13: 0000000000000036 R14: 0000000000000000 R15: 000000000009fb52
[ 9371.401403] FS:  00007fc9849f2980(0000) GS:ffff88942fb80000(0000) knlGS:0000000000000000
[ 9371.410244] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9371.417007] CR2: 00007f20d02c3000 CR3: 0000000440b9c000 CR4: 00000000003406e0
[ 9371.424889] Call Trace:
[ 9371.428054]  <IRQ>
[ 9371.436315]  save_stack+0x3f/0x50
[ 9371.734034]  kasan_slab_free+0xe/0x10
[ 9371.738798]  slab_free_freelist_hook+0x5d/0x1c0
[ 9371.748886]  kmem_cache_free+0x10c/0x390
[ 9371.758450]  mempool_free_slab+0x17/0x20
[ 9371.763522]  mempool_free+0x65/0x170
[ 9371.767825]  bio_free+0x14c/0x210
[ 9371.771864]  bio_put+0x59/0x70
[ 9371.775644]  end_swap_bio_write+0x199/0x250
[ 9371.780556]  bio_endio+0x22c/0x4e0
[ 9371.784709]  dec_pending+0x1bf/0x3e0 [dm_mod]
[ 9371.790207]  clone_endio+0x129/0x3d0 [dm_mod]
[ 9371.800746]  bio_endio+0x22c/0x4e0
[ 9371.809262]  blk_update_request+0x3bb/0x980
[ 9371.814605]  scsi_end_request+0x53/0x420
[ 9371.824002]  scsi_io_completion+0x10a/0x830
[ 9371.844445]  scsi_finish_command+0x1b9/0x250
[ 9371.849445]  scsi_softirq_done+0x1ab/0x1f0
[ 9371.854272]  blk_mq_force_complete_rq+0x217/0x250
[ 9371.859708]  blk_mq_complete_request+0xe/0x20
[ 9371.865171]  scsi_mq_done+0xc1/0x220
[ 9371.869479]  pqi_aio_io_complete+0x83/0x2c0 [smartpqi]
[ 9371.881764]  pqi_irq_handler+0x1fc/0x13f0 [smartpqi]
[ 9371.914115]  __handle_irq_event_percpu+0x81/0x550
[ 9371.924289]  handle_irq_event_percpu+0x70/0x100
[ 9371.945559]  handle_irq_event+0x5a/0x8b
[ 9371.950121]  handle_edge_irq+0x10c/0x370
[ 9371.950121]  handle_edge_irq+0x10c/0x370
[ 9371.959858]  asm_call_on_stack+0x12/0x20
asm_call_on_stack at arch/x86/entry/entry_64.S:710
[ 9371.964899]  </IRQ>
[ 9371.967716]  common_interrupt+0x185/0x2a0
[ 9371.972455]  asm_common_interrupt+0x1e/0x40
[ 9371.977368] RIP: 0010:__asan_load4+0x8/0xa0
[ 9371.982281] Code: 00 e8 5c f4 ff ff 5d c3 40 38 f0 0f 9e c0 84 c0 75 e5 5d c3 48 c1 e8 03 80 3c 10 00 75 ed 5d c3 66 90 55 48 89 e5 48 8b 4d 08 <48> 83 ff fb 77 6c eb 3a 0f 1f 00 48 b8 00 00 00 00 00 00 00 ff 48
[ 9372.002246] RSP: 0018:ffffc9000b12f0e8 EFLAGS: 00000202
[ 9372.008207] RAX: 0000000000000000 RBX: ffffc9000b12f150 RCX: ffffffff9ed32487
[ 9372.016489] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffffffff9ff92a94
[ 9372.024370] RBP: ffffc9000b12f0e8 R08: fffffbfff3ff1d4d R09: fffffbfff3ff1d4d
[ 9372.032252] R10: ffffffff9ff8ea67 R11: fffffbfff3ff1d4c R12: 0000000000000000
[ 9372.040490] R13: ffffc9000b12f358 R14: 000000000000000c R15: 0000000000000005
[ 9372.053899]  debug_lockdep_rcu_enabled+0x27/0x60
[ 9372.059248]  rcu_read_lock_held_common+0x12/0x60
[ 9372.064989]  rcu_read_lock_sched_held+0x60/0xe0
[ 9372.080338]  shrink_active_list+0xbfd/0xc30
[ 9372.120481]  shrink_lruvec+0xbf1/0x11b0
[ 9372.150410]  shrink_node+0x344/0xd10
[ 9372.154719]  do_try_to_free_pages+0x263/0xa00
[ 9372.169860]  try_to_free_pages+0x239/0x570
[ 9372.179356]  __alloc_pages_slowpath.constprop.59+0x5dd/0x1880
[ 9372.215762]  __alloc_pages_nodemask+0x562/0x670
[ 9372.232686]  alloc_pages_current+0x9c/0x110
[ 9372.237599]  alloc_slab_page+0x355/0x530
[ 9372.242755]  allocate_slab+0x485/0x5a0
[ 9372.247232]  new_slab+0x46/0x70
[ 9372.251095]  ___slab_alloc+0x35f/0x810
[ 9372.274485]  __slab_alloc+0x43/0x70
[ 9372.287650]  kmem_cache_alloc+0x257/0x3d0
[ 9372.298182]  prepare_creds+0x26/0x130
[ 9372.302571]  do_faccessat+0x255/0x3e0
[ 9372.321591]  __x64_sys_access+0x38/0x40
[ 9372.326154]  do_syscall_64+0x64/0x340
[ 9372.330542]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9372.336329] RIP: 0033:0x7fc983a21bfb
[ 9372.341050] Code: Bad RIP value.
[ 9372.345000] RSP: 002b:00007fffd543cde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000015
[ 9372.353319] RAX: ffffffffffffffda RBX: 00007fffd543fa40 RCX: 00007fc983a21bfb
[ 9372.361199] RDX: 00007fc983cf1c00 RSI: 0000000000000000 RDI: 00005573ed458090
[ 9372.369512] RBP: 00007fffd543cf30 R08: 00005573ed44ab99 R09: 0000000000000007
[ 9372.377393] R10: 0000000000000041 R11: 0000000000000246 R12: 0000000000000000
[ 9372.385275] R13: 0000000000000000 R14: 00007fffd543cea0 R15: 00005573eecd2990
[ 9372.393608] irq event stamp: 27274496
[ 9372.397999] hardirqs last  enabled at (27274495): [<ffffffff9e635b8e>] free_unref_page_list+0x2ee/0x400
[ 9372.408152] hardirqs last disabled at (27274496): [<ffffffff9ed2c22b>] idtentry_enter_cond_rcu+0x1b/0x50
[ 9372.418695] softirqs last  enabled at (27272816): [<ffffffff9f000478>] __do_softirq+0x478/0x784
[ 9372.428154] softirqs last disabled at (27272807): [<ffffffff9e2d0b41>] irq_exit_rcu+0xd1/0xe0
[ 9372.437435] ---[ end trace d2ebac1fad6e452e ]---
#30 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

CC + Alexander
Quoted 8 lines · Qian Cai
Quoted 2 lines · Peter Zijlstra
Quoted 2 lines
[ 9371.959858]  asm_call_on_stack+0x12/0x20

asm_call_on_stack at arch/x86/entry/entry_64.S:710

This is one piece of call from the warning call traces that introduced by the patch which leads me to revert the commit in the first place. It may or may not be the real culprit, but just wanted to highlight it in case.

Oh well. The warning is a storage check in the stack depot code, i.e. stack depot ran out of storage space.

Even if that commit causes stack traces to be larger that revert does not make any sense at all and handwaving about recursions does not help either. If that commit introduced a recursion then that would have worse effects than triggering this warning.

The difference between the interrupt stack switching introduced by this commit is that it generates another entry in the stack trace compared to the state before it, which obviously has an effect on the storage requirements in the stack depot.

Thanks,

tglx

#31 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 21 lines · Thomas Gleixner
CC + Alexander
Quoted 8 lines · Qian Cai
Quoted 2 lines · Peter Zijlstra
Quoted 2 lines
[ 9371.959858]  asm_call_on_stack+0x12/0x20

asm_call_on_stack at arch/x86/entry/entry_64.S:710

This is one piece of call from the warning call traces that introduced by the patch which leads me to revert the commit in the first place. It may or may not be the real culprit, but just wanted to highlight it in case.

Oh well. The warning is a storage check in the stack depot code, i.e. stack depot ran out of storage space.

Even if that commit causes stack traces to be larger that revert does not make any sense at all and handwaving about recursions does not help either. If that commit introduced a recursion then that would have worse effects than triggering this warning.

Quoted 4 lines

The difference between the interrupt stack switching introduced by this commit is that it generates another entry in the stack trace compared to the state before it, which obviously has an effect on the storage requirements in the stack depot.

Sorry, not meant to propose the reverting as a final solution to the problem, but rather a data point. My problem is that I knew so little about this kernel area, so I was to throw out something where I saw something similiar while working in other areas over the years. Of course, my "random" tips could be wrong.

The effect is quite bad for our CI because it will disable lockdep in the middle of the tests which could miss regression bugs lockdep might find later. Anyway, I am not motivate enough to measure how much "an effect" is yet due to this patchset. I'll trim our .config and tests to hopefully not running out of the storage this early.

#32 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

In reply to Peter Zijlstra
Quoted 37 lines · Peter Zijlstra
Quoted 36 lines · Qian Cai
Quoted 23 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.

As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.

That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.

The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.

Quoted 8 lines

Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.

64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.

Reverted this commit and the rest of series (with trivial fixup) as well as the two dependencies [1],

Quoted 9 lines
Quoted 4 lines

8449e768dcb8 ("x86/entry: Remove debug IDT frobbing") 029149180d1d ("x86/entry: Rename trace_hardirqs_off_prepare()")

fixed the warning under some memory pressure on AMD NUMA servers.

The warning ?

Quoted 2 lines
[ 9371.959858]  asm_call_on_stack+0x12/0x20

asm_call_on_stack at arch/x86/entry/entry_64.S:710

Even after I trimmed the .config [1] to the barely minimal, a subset of LTP test still unable to finish on those AMD servers with page_owner=on.

[1]:
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config

It looks like because this new IRQ entry introduced by this patch,

</IRQ>

asm_call_on_stack at arch/x86/entry/entry_64.S:710 handle_edge_irq at kernel/irq/chip.c:832

which will running out of the stack depot limit due to nested loops below.

which has this loop,

	do {
		...
		handle_irq_event(desc);
		...
	} while ((desc->istate & IRQS_PENDING) &&
		!irqd_irq_disabled(&desc->irq_data));

handle_irq_event at kernel/irq/handle.c:215 __handle_irq_event_percpu at kernel/irq/handle.c:156

Here has a nested loop,

	for_each_action_of_desc(desc, action) {
		...
		res = action->handler(irq, action->dev_id);
		...
	}

pqi_irq_handler at drivers/scsi/smartpqi/smartpqi_init.c:3462 pqi_process_io_intr at drivers/scsi/smartpqi/smartpqi_init.c:2992

Here has another nested loop,

	while (1) {
		...
		io_request->io_complete_callback(io_request,
			io_request->context);
		...
	}

scsi_mq_done at drivers/scsi/scsi_lib.c:1602
blk_mq_complete_request at block/blk-mq.c:677
blk_mq_force_complete_rq at block/blk-mq.c:637
scsi_io_completion at drivers/scsi/scsi_lib.c:934
scsi_end_request at drivers/scsi/scsi_lib.c:584
scsi_mq_uninit_cmd at drivers/scsi/scsi_lib.c:547
scsi_free_sgtables at drivers/scsi/scsi_lib.c:537
__sg_free_table at lib/scatterlist.c:225

Here has one more nested loop,

	while (table->orig_nents) {
		...
		free_fn(sgl, alloc_size);
		...
	}

free_fn() will call kmem_cache_free(). Since we have page_owner=on, it will call save_stack() to save the each free stack trace.

Quoted 106 lines
Quoted 4 lines
[ 9371.260161] ------------[ cut here ]------------
[ 9371.267143] Stack depot reached limit capacity
[ 9371.267193] WARNING: CPU: 19 PID: 1181 at lib/stackdepot.c:115 stack_depot_save+0x3d9/0x57d

Is this _the_ warning?

Maybe it really is running out of storage, I don't think this patch is to blame. The unwind here looks good, so why would the stack-depot unwind be any different?

Quoted 95 lines
[ 9371.281470] Modules linked in: brd vfat fat ext4 crc16 mbcache jbd2 loop kvm_amd kvm ses enclosure dax_pmem irqbypass dax_pmem_core acpi_cpufreq ip_tables x_table
s xfs sd_mod bnxt_en smartpqi scsi_transport_sas tg3 i40e libphy firmware_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dummy_del_mod]
[ 9371.310176] CPU: 19 PID: 1181 Comm: systemd-journal Tainted: G           O      5.7.0-next-20200604+ #1
[ 9371.320700] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 03/09/2018
[ 9371.329987] RIP: 0010:stack_depot_save+0x3d9/0x57d
[ 9371.335513] Code: 1d 9b bc 68 01 80 fb 01 0f 87 c0 01 00 00 80 e3 01 75 1f 4c 89 45 c0 c6 05 82 bc 68 01 01 48 c7 c7 e0 85 63 9f e8 9d 74 9d ff <0f> 0b 90 90 4c 8
b 45 c0 48 c7 c7 80 1a d0 9f 4c 89 c6 e8 b0 2b 46
[ 9371.355426] RSP: 0018:ffffc90007260490 EFLAGS: 00010082
[ 9371.361387] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff9ed3207f
[ 9371.369544] RDX: 0000000000000007 RSI: dffffc0000000000 RDI: 0000000000000000
[ 9371.377428] RBP: ffffc900072604f8 R08: fffffbfff3f37539 R09: fffffbfff3f37539
[ 9371.385310] R10: ffffffff9f9ba9c3 R11: fffffbfff3f37538 R12: ffffc90007260508
[ 9371.393521] R13: 0000000000000036 R14: 0000000000000000 R15: 000000000009fb52
[ 9371.401403] FS:  00007fc9849f2980(0000) GS:ffff88942fb80000(0000) knlGS:0000000000000000
[ 9371.410244] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9371.417007] CR2: 00007f20d02c3000 CR3: 0000000440b9c000 CR4: 00000000003406e0
[ 9371.424889] Call Trace:
[ 9371.428054]  <IRQ>
[ 9371.436315]  save_stack+0x3f/0x50
[ 9371.734034]  kasan_slab_free+0xe/0x10
[ 9371.738798]  slab_free_freelist_hook+0x5d/0x1c0
[ 9371.748886]  kmem_cache_free+0x10c/0x390
[ 9371.758450]  mempool_free_slab+0x17/0x20
[ 9371.763522]  mempool_free+0x65/0x170
[ 9371.767825]  bio_free+0x14c/0x210
[ 9371.771864]  bio_put+0x59/0x70
[ 9371.775644]  end_swap_bio_write+0x199/0x250
[ 9371.780556]  bio_endio+0x22c/0x4e0
[ 9371.784709]  dec_pending+0x1bf/0x3e0 [dm_mod]
[ 9371.790207]  clone_endio+0x129/0x3d0 [dm_mod]
[ 9371.800746]  bio_endio+0x22c/0x4e0
[ 9371.809262]  blk_update_request+0x3bb/0x980
[ 9371.814605]  scsi_end_request+0x53/0x420
[ 9371.824002]  scsi_io_completion+0x10a/0x830
[ 9371.844445]  scsi_finish_command+0x1b9/0x250
[ 9371.849445]  scsi_softirq_done+0x1ab/0x1f0
[ 9371.854272]  blk_mq_force_complete_rq+0x217/0x250
[ 9371.859708]  blk_mq_complete_request+0xe/0x20
[ 9371.865171]  scsi_mq_done+0xc1/0x220
[ 9371.869479]  pqi_aio_io_complete+0x83/0x2c0 [smartpqi]
[ 9371.881764]  pqi_irq_handler+0x1fc/0x13f0 [smartpqi]
[ 9371.914115]  __handle_irq_event_percpu+0x81/0x550
[ 9371.924289]  handle_irq_event_percpu+0x70/0x100
[ 9371.945559]  handle_irq_event+0x5a/0x8b
[ 9371.950121]  handle_edge_irq+0x10c/0x370
[ 9371.950121]  handle_edge_irq+0x10c/0x370
[ 9371.959858]  asm_call_on_stack+0x12/0x20
asm_call_on_stack at arch/x86/entry/entry_64.S:710
[ 9371.964899]  </IRQ>
[ 9371.967716]  common_interrupt+0x185/0x2a0
[ 9371.972455]  asm_common_interrupt+0x1e/0x40
[ 9371.977368] RIP: 0010:__asan_load4+0x8/0xa0
[ 9371.982281] Code: 00 e8 5c f4 ff ff 5d c3 40 38 f0 0f 9e c0 84 c0 75 e5 5d c3 48 c1 e8 03 80 3c 10 00 75 ed 5d c3 66 90 55 48 89 e5 48 8b 4d 08 <48> 83 ff fb 77 6c eb 3a 0f 1f 00 48 b8 00 00 00 00 00 00 00 ff 48
[ 9372.002246] RSP: 0018:ffffc9000b12f0e8 EFLAGS: 00000202
[ 9372.008207] RAX: 0000000000000000 RBX: ffffc9000b12f150 RCX: ffffffff9ed32487
[ 9372.016489] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffffffff9ff92a94
[ 9372.024370] RBP: ffffc9000b12f0e8 R08: fffffbfff3ff1d4d R09: fffffbfff3ff1d4d
[ 9372.032252] R10: ffffffff9ff8ea67 R11: fffffbfff3ff1d4c R12: 0000000000000000
[ 9372.040490] R13: ffffc9000b12f358 R14: 000000000000000c R15: 0000000000000005
[ 9372.053899]  debug_lockdep_rcu_enabled+0x27/0x60
[ 9372.059248]  rcu_read_lock_held_common+0x12/0x60
[ 9372.064989]  rcu_read_lock_sched_held+0x60/0xe0
[ 9372.080338]  shrink_active_list+0xbfd/0xc30
[ 9372.120481]  shrink_lruvec+0xbf1/0x11b0
[ 9372.150410]  shrink_node+0x344/0xd10
[ 9372.154719]  do_try_to_free_pages+0x263/0xa00
[ 9372.169860]  try_to_free_pages+0x239/0x570
[ 9372.179356]  __alloc_pages_slowpath.constprop.59+0x5dd/0x1880
[ 9372.215762]  __alloc_pages_nodemask+0x562/0x670
[ 9372.232686]  alloc_pages_current+0x9c/0x110
[ 9372.237599]  alloc_slab_page+0x355/0x530
[ 9372.242755]  allocate_slab+0x485/0x5a0
[ 9372.247232]  new_slab+0x46/0x70
[ 9372.251095]  ___slab_alloc+0x35f/0x810
[ 9372.274485]  __slab_alloc+0x43/0x70
[ 9372.287650]  kmem_cache_alloc+0x257/0x3d0
[ 9372.298182]  prepare_creds+0x26/0x130
[ 9372.302571]  do_faccessat+0x255/0x3e0
[ 9372.321591]  __x64_sys_access+0x38/0x40
[ 9372.326154]  do_syscall_64+0x64/0x340
[ 9372.330542]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9372.336329] RIP: 0033:0x7fc983a21bfb
[ 9372.341050] Code: Bad RIP value.
[ 9372.345000] RSP: 002b:00007fffd543cde8 EFLAGS: 00000246 ORIG_RAX: 0000000000000015
[ 9372.353319] RAX: ffffffffffffffda RBX: 00007fffd543fa40 RCX: 00007fc983a21bfb
[ 9372.361199] RDX: 00007fc983cf1c00 RSI: 0000000000000000 RDI: 00005573ed458090
[ 9372.369512] RBP: 00007fffd543cf30 R08: 00005573ed44ab99 R09: 0000000000000007
[ 9372.377393] R10: 0000000000000041 R11: 0000000000000246 R12: 0000000000000000
[ 9372.385275] R13: 0000000000000000 R14: 00007fffd543cea0 R15: 00005573eecd2990
[ 9372.393608] irq event stamp: 27274496
[ 9372.397999] hardirqs last  enabled at (27274495): [<ffffffff9e635b8e>] free_unref_page_list+0x2ee/0x400
[ 9372.408152] hardirqs last disabled at (27274496): [<ffffffff9ed2c22b>] idtentry_enter_cond_rcu+0x1b/0x50
[ 9372.418695] softirqs last  enabled at (27272816): [<ffffffff9f000478>] __do_softirq+0x478/0x784
[ 9372.428154] softirqs last disabled at (27272807): [<ffffffff9e2d0b41>] irq_exit_rcu+0xd1/0xe0
[ 9372.437435] ---[ end trace d2ebac1fad6e452e ]---
#33 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Qian,

can you please ensure that people who got cc'ed because the problem affects their subsystem are included on your replies even if you are replying to a different subthread?

I explicitely did:

Cc + Alexander

at the very beginning of my reply:

https://lore.kernel.org/r/[email protected]

to make you aware of that.

Yes, email sucks, but it sucks even more when people are careless.

Quoted 4 lines · Qian Cai

On Fri, Jun 05, 2020 at 07:36:22PM +0200, Peter Zijlstra wrote:

Even after I trimmed the .config [1] to the barely minimal, a subset of LTP test still unable to finish on those AMD servers with page_owner=on.

What a surprise...

Quoted 12 lines

[1]:
https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config

It looks like because this new IRQ entry introduced by this patch,

</IRQ>

asm_call_on_stack at arch/x86/entry/entry_64.S:710 handle_edge_irq at kernel/irq/chip.c:832

which will running out of the stack depot limit due to nested loops below.

Quoted 8 lines

which has this loop,

	do {
		...
		handle_irq_event(desc);
		...
	} while ((desc->istate & IRQS_PENDING) &&
		!irqd_irq_disabled(&desc->irq_data));

This loop has absolutely nothing to do with stack entry usage.

foo()

  do {
     bar();

} while (condition);
}

If you take a stack trace inside bar() it will be the same stack trace for every single loop iteration. And that stack trace will not be any different from:

foo()
{
bar():
}

assumed that the call chain leading to foo() is the same in both cases.

And you can add even more loops in subsequent call chains within bar(). They do not matter at all.

Quoted 7 lines

Here has a nested loop,

	    for_each_action_of_desc(desc, action) {
		    ...
		    res = action->handler(irq, action->dev_id);
		    ...
	    }

And this one is completely irrelevant because the interrupt which we are looking at is a PCI interrupt which CANNOT be shared. IOW, the number of loop iterations and the number of handlers invoked is exactly ONE.

I seriously have no idea what you are trying to demonstrate by finding loops in a SINGLE callchain.

Quoted 10 lines

Here has one more nested loop,

	while (table->orig_nents) {
		...
		free_fn(sgl, alloc_size);
		...
	}

free_fn() will call kmem_cache_free(). Since we have page_owner=on, it will call save_stack() to save the each free stack trace.

That stack trace for each invocation of free_fn() in this loop is exactly the same stack trace. The same stack trace is not eating up any memory because the hash matches, i.e. the stack trace in the depot is already known.

Here is the simplified difference between the old code and the new code:

Old New

handle_edge_irq handle_edge_irq
do_IRQ asm_call_on_stack
common_interrupt common_interrupt
asm_common_interrupt

IOW, for every _UNIQUE_ interrupt related call chain, there is exactly ONE stack entry more than before.

For a loop which generates the exact same stack trace for every iteration this extra entry is not a problem.

But what matters is that interrupts can hit any random code path. So the amount of possible non-unique call chains is pretty much unlimited. And with a high number of non-unique call chains the extra entry starts to matter.

It's trival math, isn't it?

TS  = Total Size of depot
as  = average size of all stored unique stack traces
ms  = maximum number of unqiue stack traces which fit in TS

ms_old = TS / as

Lets further assume that the vast majority of stack traces are taken from interrupt context. That means with the new code this results in:

    ms_new = TS / (as + 1)

==> ms_new = ms_old * as / (as + 1)

Depending on the value of 'as' the +1 can shave off a significant percentage of capacity. IOW, the capacity is simply too small now for the test scenario you are running. Truly a surprising outcome, right?

To get facts instead of useless loop theories, can you please apply the patch below, enable DEBUGFS and provide the output of

/sys/kernel/debug/stackdepot/info

for a kernel before that change and after? Please read out that file at periodically roughly the same amounts of time after starting your test scenario.

Note, that I doubled the size of the stack depot so that we get real numbers and not the cutoff by the size limit. IOW, the warning should not trigger anymore. If it triggers nevertheless then the numbers will still tell us an interesting story.

Thanks,

tglx

---
 lib/stackdepot.c |   43 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 42 insertions(+), 1 deletion(-)
lib/stackdepot.c +42 -1
@@ -31,6 +31,7 @@
 #include <linux/stackdepot.h>
 #include <linux/string.h>
 #include <linux/types.h>
+#include <linux/debugfs.h>
 
 #define DEPOT_STACK_BITS (sizeof(depot_stack_handle_t) * 8)
 
@@ -42,7 +43,7 @@
 					STACK_ALLOC_ALIGN)
 #define STACK_ALLOC_INDEX_BITS (DEPOT_STACK_BITS - \
 		STACK_ALLOC_NULL_PROTECTION_BITS - STACK_ALLOC_OFFSET_BITS)
-#define STACK_ALLOC_SLABS_CAP 8192
+#define STACK_ALLOC_SLABS_CAP 16384
 #define STACK_ALLOC_MAX_SLABS \
 	(((1LL << (STACK_ALLOC_INDEX_BITS)) < STACK_ALLOC_SLABS_CAP) ? \
 	 (1LL << (STACK_ALLOC_INDEX_BITS)) : STACK_ALLOC_SLABS_CAP)
@@ -70,6 +71,7 @@ static void *stack_slabs[STACK_ALLOC_MAX
 static int depot_index;
 static int next_slab_inited;
 static size_t depot_offset;
+static unsigned long unique_stacks;
 static DEFINE_SPINLOCK(depot_lock);
 
 static bool init_stack_slab(void **prealloc)
@@ -138,6 +140,7 @@ static struct stack_record *depot_alloc_
 	stack->handle.valid = 1;
 	memcpy(stack->entries, entries, size * sizeof(unsigned long));
 	depot_offset += required_size;
+	unique_stacks++;
 
 	return stack;
 }
@@ -340,3 +343,41 @@ unsigned int filter_irq_stacks(unsigned
 	return nr_entries;
 }
 EXPORT_SYMBOL_GPL(filter_irq_stacks);
+
+static int debug_show(struct seq_file *m, void *p)
+{
+	unsigned long unst;
+	int didx, doff;
+
+	spin_lock_irq(&depot_lock);
+	unst = unique_stacks;
+	didx = depot_index;
+	doff = depot_offset;
+	spin_unlock_irq(&depot_lock);
+
+	seq_printf(m, "Unique stacks: %lu\n", unst);
+	seq_printf(m, "Depot index:   %d\n", didx);
+	seq_printf(m, "Depot offset:  %d\n", doff);
+	return 0;
+}
+
+static int debug_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, debug_show, inode->i_private);
+}
+
+static const struct file_operations dfs_ops = {
+	.open		= debug_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
+static int __init debugfs_init(void)
+{
+	struct dentry *root_dir = debugfs_create_dir("stackdepot", NULL);
+
+	debugfs_create_file("info", 0444, root_dir, NULL, &dfs_ops);
+	return 0;
+}
+__initcall(debugfs_init);
#34 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 10 lines · Thomas Gleixner

Qian,

can you please ensure that people who got cc'ed because the problem affects their subsystem are included on your replies even if you are replying to a different subthread?

I explicitely did:

Cc + Alexander
Quoted 7 lines

at the very beginning of my reply:

   https://lore.kernel.org/r/[email protected]

to make you aware of that.

Yes, email sucks, but it sucks even more when people are careless.

Sorry, I will remeber that next time.

[]

Quoted 5 lines

To get facts instead of useless loop theories, can you please apply the patch below, enable DEBUGFS and provide the output of

/sys/kernel/debug/stackdepot/info

Quoted 8 lines

for a kernel before that change and after? Please read out that file at periodically roughly the same amounts of time after starting your test scenario.

Note, that I doubled the size of the stack depot so that we get real numbers and not the cutoff by the size limit. IOW, the warning should not trigger anymore. If it triggers nevertheless then the numbers will still tell us an interesting story.

Instead of running the whole testsuite, I just picked this single LTP oom02 test which seems usually trigger it within the testsuite. Let me know if this is insufficient (which indeed tell the big difference in "Unique stacks"), and I am happy to run the whole things.

BAD: next-20200608
GOOD: next-20200528 (which does not include this series)

BAD (after boot)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 33547
Depot index:   359
Depot offset:  6752

BAD (after oom02)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 140476
Depot index:   2555
Depot offset:  9168

GOOD (after boot)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 31112
Depot index:   317
Depot offset:  14384

GOOD (after oom02)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 34176
Depot index:   354
Depot offset:  4032

BTW, I am happy to run another one using next-20200608 with just this series reverted if you suspect there is something else going on between those two trees.

#35 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Qian,

Quoted 10 lines · Qian Cai
Quoted 4 lines · Thomas Gleixner

Note, that I doubled the size of the stack depot so that we get real numbers and not the cutoff by the size limit. IOW, the warning should not trigger anymore. If it triggers nevertheless then the numbers will still tell us an interesting story.

Instead of running the whole testsuite, I just picked this single LTP oom02 test which seems usually trigger it within the testsuite. Let me know if this is insufficient (which indeed tell the big difference in "Unique stacks"), and I am happy to run the whole things.

thanks for providing the data.

Quoted 4 lines

BAD: next-20200608
GOOD: next-20200528 (which does not include this series)

BAD (after boot)

Quoted 8 lines
# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 33547
Depot index:   359
Depot offset:  6752

BAD (after oom02)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 140476

That's indeed odd. I try to reproduce and figure out what really breaks here.

Thanks,

tglx

#36 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Qian,

Quoted 6 lines · Thomas Gleixner
Quoted 3 lines

BAD (after oom02)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 140476

That's indeed odd. I try to reproduce and figure out what really breaks here.

I checked your config file and I think I know where this comes from. Can you plase disable KASAN just for testing purposes and compare before after again?

Thanks,

tglx

#37 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 7 lines · Thomas Gleixner

Qian,

Quoted 4 lines · Thomas Gleixner
Quoted 3 lines

BAD (after oom02)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 140476
Quoted 6 lines
Quoted 2 lines

That's indeed odd. I try to reproduce and figure out what really breaks here.

I checked your config file and I think I know where this comes from. Can you plase disable KASAN just for testing purposes and compare before after again?

It turns out I'll need a few days to be able to get ahold of those affected systems again. I'll be reporting back as soon as possible.

#38 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 9 lines · Qian Cai
Quoted 8 lines · Thomas Gleixner
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines

BAD (after oom02)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 140476

That's indeed odd. I try to reproduce and figure out what really breaks here.

Quoted 6 lines
Quoted 3 lines

I checked your config file and I think I know where this comes from. Can you plase disable KASAN just for testing purposes and compare before after again?

It turns out I'll need a few days to be able to get ahold of those affected systems again. I'll be reporting back as soon as possible.

I figured it out. It has nothing to do with asm_call_on...(). It's also unrelated to page_owner. It's purely a KASAN issue.

KASAN trims stack traces with the help of filter_irq_stacks() which relies on __irqentry_text_start/end

The entry rework moved the interrupt entry points out of the irqentry section, which breaks that filter function.

I made it work again. Here are the numbers which make that obvious:

Mainline:

Unique stacks: 23403
Depot index:   244
Depot offset:  4208

x86/entry:

Unique stacks: 38835
Depot index:   464
Depot offset:  7616

x86/entry + fix:

Unique stacks: 23607
Depot index:   247
Depot offset:  14224

So with the non-working trimming this generates more unique stacks and because they are not trimmed they become larger and eat more storage space. The resulting average per stack shows that:

Mainline:           171 bytes per stack
x86/entry:          195 bytes per stack

x86/entry + fix: 172 bytes per stack

I'll point you to a test branch shortly.

Thanks,

tglx

#39 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 47 lines · Thomas Gleixner
Quoted 9 lines · Qian Cai
Quoted 8 lines · Thomas Gleixner
Quoted 6 lines · Thomas Gleixner
Quoted 3 lines

BAD (after oom02)

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 140476

That's indeed odd. I try to reproduce and figure out what really breaks here.

Quoted 6 lines
Quoted 3 lines

I checked your config file and I think I know where this comes from. Can you plase disable KASAN just for testing purposes and compare before after again?

It turns out I'll need a few days to be able to get ahold of those affected systems again. I'll be reporting back as soon as possible.

I figured it out. It has nothing to do with asm_call_on...(). It's also unrelated to page_owner. It's purely a KASAN issue.

KASAN trims stack traces with the help of filter_irq_stacks() which relies on __irqentry_text_start/end

The entry rework moved the interrupt entry points out of the irqentry section, which breaks that filter function.

I made it work again. Here are the numbers which make that obvious:

Mainline:

Unique stacks: 23403
Depot index:   244
Depot offset:  4208

x86/entry:

Unique stacks: 38835
Depot index:   464
Depot offset:  7616

x86/entry + fix:

Unique stacks: 23607
Depot index:   247
Depot offset:  14224

So with the non-working trimming this generates more unique stacks and because they are not trimmed they become larger and eat more storage space. The resulting average per stack shows that:

Quoted 5 lines
Mainline:           171 bytes per stack
x86/entry:          195 bytes per stack

x86/entry + fix: 172 bytes per stack

I'll point you to a test branch shortly.

Thomas, I get ahold of one of the affected systems now if you want some testing there.

#40 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Qian,

Quoted 4 lines · Qian Cai

On Wed, Jun 10, 2020 at 09:38:56PM +0200, Thomas Gleixner wrote:

Thomas, I get ahold of one of the affected systems now if you want some testing there.

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry

Thanks,

tglx

#41 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 9 lines · Thomas Gleixner

Qian,

Quoted 4 lines · Qian Cai

On Wed, Jun 10, 2020 at 09:38:56PM +0200, Thomas Gleixner wrote:

Thomas, I get ahold of one of the affected systems now if you want some testing there.

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry

Looks good.

<start>

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 27481
Depot index:   285
Depot offset:  3024

<after oom02>

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 29242
Depot index:   308
Depot offset:  6608

<after the full testsuite>

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 49221
Depot index:   530
Depot offset:  5888
#42 lore raw

Re: [patch V9 10/39] x86/entry: Provide helpers for execute on irqstack

Quoted 25 lines · Qian Cai
Quoted 9 lines · Thomas Gleixner

Qian,

Quoted 4 lines · Qian Cai

On Wed, Jun 10, 2020 at 09:38:56PM +0200, Thomas Gleixner wrote:

Thomas, I get ahold of one of the affected systems now if you want some testing there.

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/entry

Looks good.

<start>

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 27481
Depot index:   285
Depot offset:  3024

<after oom02>

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 29242
Depot index:   308
Depot offset:  6608
Quoted 5 lines

<after the full testsuite>

# cat /sys/kernel/debug/stackdepot/info
Unique stacks: 49221
Depot index:   530
Depot offset:  5888

Thanks for confirming!

#43 lore raw

PATCH v9 11/39 x86/entry/64: Move do_softirq_own_stack() to C

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

The first step to get rid of the ENTER/LEAVE_IRQ_STACK ASM macro maze. Use the new C code helpers to move do_softirq_own_stack() out of ASM code.

---
V9: Simplified version

---
 arch/x86/entry/entry_64.S |   13 -------------
 arch/x86/kernel/irq_64.c  |    6 ++++++
 2 files changed, 6 insertions(+), 13 deletions(-)
arch/x86/entry/entry_64.S +0 -13
@@ -1145,19 +1145,6 @@ SYM_FUNC_START(asm_call_on_stack)
 	ret
 SYM_FUNC_END(asm_call_on_stack)
 
-/* Call softirq on interrupt stack. Interrupts are off. */
-.pushsection .text, "ax"
-SYM_FUNC_START(do_softirq_own_stack)
-	pushq	%rbp
-	mov	%rsp, %rbp
-	ENTER_IRQ_STACK regs=0 old_rsp=%r11
-	call	__do_softirq
-	LEAVE_IRQ_STACK regs=0
-	leaveq
-	ret
-SYM_FUNC_END(do_softirq_own_stack)
-.popsection
-
 #ifdef CONFIG_XEN_PV
 /*
  * A note on the "critical region" in our callback handler.
arch/x86/kernel/irq_64.c +6 -0
@@ -20,6 +20,7 @@
 #include <linux/sched/task_stack.h>
 
 #include <asm/cpu_entry_area.h>
+#include <asm/irq_stack.h>
 #include <asm/io_apic.h>
 #include <asm/apic.h>
 
@@ -70,3 +71,8 @@ int irq_init_percpu_irqstack(unsigned in
 		return 0;
 	return map_irq_stack(cpu);
 }
+
+void do_softirq_own_stack(void)
+{
+	run_on_irqstack_cond(__do_softirq, NULL, NULL);
+}
#44 lore raw

[tip: x86/entry] x86/entry/64: Move do_softirq_own_stack() to C

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     4f41db23ef2593aec26e23e7f4bacc620e82f545
Gitweb:        https://git.kernel.org/tip/4f41db23ef2593aec26e23e7f4bacc620e82f545
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:24 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

x86/entry/64: Move do_softirq_own_stack() to C

The first step to get rid of the ENTER/LEAVE_IRQ_STACK ASM macro maze. Use the new C code helpers to move do_softirq_own_stack() out of ASM code.

---
 arch/x86/entry/entry_64.S | 13 -------------
 arch/x86/kernel/irq_64.c  |  6 ++++++
 2 files changed, 6 insertions(+), 13 deletions(-)
arch/x86/entry/entry_64.S +0 -13
index 1597370..6b518be 100644
@@ -1145,19 +1145,6 @@ SYM_FUNC_START(asm_call_on_stack)
 	ret
 SYM_FUNC_END(asm_call_on_stack)
 
-/* Call softirq on interrupt stack. Interrupts are off. */
-.pushsection .text, "ax"
-SYM_FUNC_START(do_softirq_own_stack)
-	pushq	%rbp
-	mov	%rsp, %rbp
-	ENTER_IRQ_STACK regs=0 old_rsp=%r11
-	call	__do_softirq
-	LEAVE_IRQ_STACK regs=0
-	leaveq
-	ret
-SYM_FUNC_END(do_softirq_own_stack)
-.popsection
-
 #ifdef CONFIG_XEN_PV
 /*
  * A note on the "critical region" in our callback handler.
arch/x86/kernel/irq_64.c +6 -0
index 12df3a4..d5f9df4 100644
@@ -20,6 +20,7 @@
 #include <linux/sched/task_stack.h>
 
 #include <asm/cpu_entry_area.h>
+#include <asm/irq_stack.h>
 #include <asm/io_apic.h>
 #include <asm/apic.h>
 
@@ -70,3 +71,8 @@ int irq_init_percpu_irqstack(unsigned int cpu)
 		return 0;
 	return map_irq_stack(cpu);
 }
+
+void do_softirq_own_stack(void)
+{
+	run_on_irqstack_cond(__do_softirq, NULL, NULL);
+}
#45 lore raw

PATCH v9 12/39 x86/entry: Split out idtentry_exit_cond_resched()

In reply to Thomas Gleixner

The XEN PV hypercall requires the ability of conditional rescheduling when preemption is disabled because some hypercalls take ages.

Split out the rescheduling code from idtentry_exit_cond_rcu() so it can be reused for that.

---
V9: New patch

---
 arch/x86/entry/common.c |   30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)
arch/x86/entry/common.c +15 -15
@@ -580,6 +580,20 @@ bool noinstr idtentry_enter_cond_rcu(str
 	return false;
 }
 
+static void idtentry_exit_cond_resched(struct pt_regs *regs, bool may_sched)
+{
+	if (may_sched && !preempt_count()) {
+		/* Sanity check RCU and thread stack */
+		rcu_irq_exit_check_preempt();
+		if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
+			WARN_ON_ONCE(!on_thread_stack());
+		if (need_resched())
+			preempt_schedule_irq();
+	}
+	/* Covers both tracing and lockdep */
+	trace_hardirqs_on();
+}
+
 /**
  * idtentry_exit_cond_rcu - Handle return from exception with conditional RCU
  *			    handling
@@ -621,21 +635,7 @@ void noinstr idtentry_exit_cond_rcu(stru
 		}
 
 		instrumentation_begin();
-
-		/* Check kernel preemption, if enabled */
-		if (IS_ENABLED(CONFIG_PREEMPTION)) {
-			if (!preempt_count()) {
-				/* Sanity check RCU and thread stack */
-				rcu_irq_exit_check_preempt();
-				if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
-					WARN_ON_ONCE(!on_thread_stack());
-				if (need_resched())
-					preempt_schedule_irq();
-			}
-		}
-		/* Covers both tracing and lockdep */
-		trace_hardirqs_on();
-
+		idtentry_exit_cond_resched(regs, IS_ENABLED(CONFIG_PREEMPTION));
 		instrumentation_end();
 	} else {
 		/*
#46 lore raw

[tip: x86/entry] x86/entry: Split out idtentry_exit_cond_resched()

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     08f086303b4eddc61da264c2e9a8a5ee7f260604
Gitweb:        https://git.kernel.org/tip/08f086303b4eddc61da264c2e9a8a5ee7f260604
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:25 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:27 +02:00

x86/entry: Split out idtentry_exit_cond_resched()

The XEN PV hypercall requires the ability of conditional rescheduling when preemption is disabled because some hypercalls take ages.

Split out the rescheduling code from idtentry_exit_cond_rcu() so it can be reused for that.

---
 arch/x86/entry/common.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)
arch/x86/entry/common.c +15 -15
index 2a80e4e..066215a 100644
@@ -583,6 +583,20 @@ bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs)
 	return false;
 }
 
+static void idtentry_exit_cond_resched(struct pt_regs *regs, bool may_sched)
+{
+	if (may_sched && !preempt_count()) {
+		/* Sanity check RCU and thread stack */
+		rcu_irq_exit_check_preempt();
+		if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
+			WARN_ON_ONCE(!on_thread_stack());
+		if (need_resched())
+			preempt_schedule_irq();
+	}
+	/* Covers both tracing and lockdep */
+	trace_hardirqs_on();
+}
+
 /**
  * idtentry_exit_cond_rcu - Handle return from exception with conditional RCU
  *			    handling
@@ -624,21 +638,7 @@ void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit)
 		}
 
 		instrumentation_begin();
-
-		/* Check kernel preemption, if enabled */
-		if (IS_ENABLED(CONFIG_PREEMPTION)) {
-			if (!preempt_count()) {
-				/* Sanity check RCU and thread stack */
-				rcu_irq_exit_check_preempt();
-				if (IS_ENABLED(CONFIG_DEBUG_ENTRY))
-					WARN_ON_ONCE(!on_thread_stack());
-				if (need_resched())
-					preempt_schedule_irq();
-			}
-		}
-		/* Covers both tracing and lockdep */
-		trace_hardirqs_on();
-
+		idtentry_exit_cond_resched(regs, IS_ENABLED(CONFIG_PREEMPTION));
 		instrumentation_end();
 	} else {
 		/*
#47 lore raw

PATCH v9 13/39 x86/entry: Switch XEN/PV hypercall entry to IDTENTRY

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Convert the XEN/PV hypercall to IDTENTRY:

- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Remove the old prototypes

The handler stubs need to stay in ASM code as it needs corner case handling and adjustment of the stack pointer.

Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack if required.

The exit code is slightly different from the regular idtentry_exit() on non-preemptible kernels. If the hypercall is preemptible and need_resched() is set then XEN provides a preempt hypercall scheduling function.

Move this functionality into the entry code so it can use the existing idtentry functionality.

---
V9: Simplified the schedule magic

---
 arch/x86/entry/common.c         |   72 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/entry/entry_32.S       |   17 +++++----
 arch/x86/entry/entry_64.S       |   22 +++---------
 arch/x86/include/asm/idtentry.h |   13 +++++++
 arch/x86/xen/setup.c            |    4 +-
 arch/x86/xen/smp_pv.c           |    3 +
 arch/x86/xen/xen-asm_32.S       |   12 +++---
 arch/x86/xen/xen-asm_64.S       |    2 -
 arch/x86/xen/xen-ops.h          |    1 
 drivers/xen/Makefile            |    2 -
 drivers/xen/preempt.c           |   42 -----------------------
 11 files changed, 115 insertions(+), 75 deletions(-)
arch/x86/entry/common.c +72 -0
@@ -27,6 +27,9 @@
 #include <linux/syscalls.h>
 #include <linux/uaccess.h>
 
+#include <xen/xen-ops.h>
+#include <xen/events.h>
+
 #include <asm/desc.h>
 #include <asm/traps.h>
 #include <asm/vdso.h>
@@ -35,6 +38,7 @@
 #include <asm/nospec-branch.h>
 #include <asm/io_bitmap.h>
 #include <asm/syscall.h>
+#include <asm/irq_stack.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/syscalls.h>
@@ -677,3 +681,71 @@ void noinstr idtentry_exit_user(struct p
 
 	prepare_exit_to_usermode(regs);
 }
+
+#ifdef CONFIG_XEN_PV
+
+#ifndef CONFIG_PREEMPTION
+/*
+ * Some hypercalls issued by the toolstack can take many 10s of
+ * seconds. Allow tasks running hypercalls via the privcmd driver to
+ * be voluntarily preempted even if full kernel preemption is
+ * disabled.
+ *
+ * Such preemptible hypercalls are bracketed by
+ * xen_preemptible_hcall_begin() and xen_preemptible_hcall_end()
+ * calls.
+ */
+DEFINE_PER_CPU(bool, xen_in_preemptible_hcall);
+EXPORT_SYMBOL_GPL(xen_in_preemptible_hcall);
+
+/*
+ * In case of scheduling the flag must be cleared and restored after
+ * returning from schedule as the task might move to a different CPU.
+ */
+static __always_inline bool get_and_clear_inhcall(void)
+{
+	boot inhcall = __this_cpu_read(xen_in_preemptible_hcall);
+
+	__this_cpu_write(xen_in_preemptible_hcall, false);
+}
+
+static __always_inline void restore_inhcall(bool inhcall)
+{
+	__this_cpu_write(xen_in_preemptible_hcall, inhcall);
+}
+#else
+static __always_inline bool get_and_clear_inhcall(void) { return false; }
+static __always_inline void restore_inhcall(bool inhcall) { }
+#endif
+
+static void __xen_pv_evtchn_do_upcall(void)
+{
+	irq_enter_rcu();
+	inc_irq_stat(irq_hv_callback_count);
+
+	xen_hvm_evtchn_do_upcall();
+
+	irq_exit_rcu();
+}
+
+__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
+{
+	struct pt_regs *old_regs;
+	bool inhcall, rcu_exit;
+
+	rcu_exit = idtentry_enter_cond_rcu(regs);
+	old_regs = set_irq_regs(regs);
+
+	run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, NULL, regs);
+
+	set_irq_regs(old_regs);
+
+	inhcall = get_and_clear_inhcall();
+	if (inhcall && !WARN_ON_ONCE(rcu_exit)) {
+		idtentry_exit_cond_resched(regs, true);
+		restore_inhcall(inhcall);
+	} else {
+		idtentry_exit_cond_rcu(regs, rcu_exit);
+	}
+}
+#endif /* CONFIG_XEN_PV */
arch/x86/entry/entry_32.S +9 -8
@@ -1298,7 +1298,10 @@ SYM_CODE_END(native_iret)
 #endif
 
 #ifdef CONFIG_XEN_PV
-SYM_FUNC_START(xen_hypervisor_callback)
+/*
+ * See comment in entry_64.S for further explanation
+ */
+SYM_FUNC_START(exc_xen_hypervisor_callback)
 	/*
 	 * Check to see if we got the event in the critical
 	 * region in xen_iret_direct, after we've reenabled
@@ -1315,14 +1318,11 @@ SYM_FUNC_START(xen_hypervisor_callback)
 	pushl	$-1				/* orig_ax = -1 => not a system call */
 	SAVE_ALL
 	ENCODE_FRAME_POINTER
-	TRACE_IRQS_OFF
+
 	mov	%esp, %eax
-	call	xen_evtchn_do_upcall
-#ifndef CONFIG_PREEMPTION
-	call	xen_maybe_preempt_hcall
-#endif
-	jmp	ret_from_intr
-SYM_FUNC_END(xen_hypervisor_callback)
+	call	xen_pv_evtchn_do_upcall
+	jmp	handle_exception_return
+SYM_FUNC_END(exc_xen_hypervisor_callback)
 
 /*
  * Hypervisor uses this for application faults while it executes.
@@ -1464,6 +1464,7 @@ SYM_CODE_START_LOCAL_NOALIGN(handle_exce
 	movl	%esp, %eax			# pt_regs pointer
 	CALL_NOSPEC edi
 
+handle_exception_return:
 #ifdef CONFIG_VM86
 	movl	PT_EFLAGS(%esp), %eax		# mix EFLAGS and CS
 	movb	PT_CS(%esp), %al
arch/x86/entry/entry_64.S +7 -15
@@ -1067,10 +1067,6 @@ apicinterrupt IRQ_WORK_VECTOR			irq_work
 
 idtentry	X86_TRAP_PF		page_fault		do_page_fault			has_error_code=1
 
-#ifdef CONFIG_XEN_PV
-idtentry	512 /* dummy */		hypervisor_callback	xen_do_hypervisor_callback	has_error_code=0
-#endif
-
 /*
  * Reload gs selector with exception handling
  * edi:  new selector
@@ -1158,9 +1154,10 @@ SYM_FUNC_END(asm_call_on_stack)
  * So, on entry to the handler we detect whether we interrupted an
  * existing activation in its critical region -- if so, we pop the current
  * activation and restart the handler using the previous one.
+ *
+ * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs)
  */
-/* do_hypervisor_callback(struct *pt_regs) */
-SYM_CODE_START_LOCAL(xen_do_hypervisor_callback)
+SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback)
 
 /*
  * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1170,15 +1167,10 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_c
 	movq	%rdi, %rsp			/* we don't return, adjust the stack frame */
 	UNWIND_HINT_REGS
 
-	ENTER_IRQ_STACK old_rsp=%r10
-	call	xen_evtchn_do_upcall
-	LEAVE_IRQ_STACK
-
-#ifndef CONFIG_PREEMPTION
-	call	xen_maybe_preempt_hcall
-#endif
-	jmp	error_exit
-SYM_CODE_END(xen_do_hypervisor_callback)
+	call	xen_pv_evtchn_do_upcall
+
+	jmp	error_return
+SYM_CODE_END(exc_xen_hypervisor_callback)
 
 /*
  * Hypervisor uses this for application faults while it executes.
arch/x86/include/asm/idtentry.h +13 -0
@@ -337,6 +337,13 @@ static __always_inline void __##func(str
  * This avoids duplicate defines and ensures that everything is consistent.
  */
 
+/*
+ * Dummy trap number so the low level ASM macro vector number checks do not
+ * match which results in emitting plain IDTENTRY stubs without bells and
+ * whistels.
+ */
+#define X86_TRAP_OTHER		0xFFFF
+
 /* Simple exception entry points. No hardware error code */
 DECLARE_IDTENTRY(X86_TRAP_DE,		exc_divide_error);
 DECLARE_IDTENTRY(X86_TRAP_OF,		exc_overflow);
@@ -376,4 +383,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB,	debug)
 /* #DF */
 DECLARE_IDTENTRY_DF(X86_TRAP_DF,	exc_double_fault);
 
+#ifdef CONFIG_XEN_PV
+DECLARE_IDTENTRY(X86_TRAP_OTHER,	exc_xen_hypervisor_callback);
+#endif
+
+#undef X86_TRAP_OTHER
+
 #endif
arch/x86/xen/setup.c +3 -1
@@ -20,6 +20,7 @@
 #include <asm/setup.h>
 #include <asm/acpi.h>
 #include <asm/numa.h>
+#include <asm/idtentry.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
 
@@ -993,7 +994,8 @@ static void __init xen_pvmmu_arch_setup(
 	HYPERVISOR_vm_assist(VMASST_CMD_enable,
 			     VMASST_TYPE_pae_extended_cr3);
 
-	if (register_callback(CALLBACKTYPE_event, xen_hypervisor_callback) ||
+	if (register_callback(CALLBACKTYPE_event,
+			      xen_asm_exc_xen_hypervisor_callback) ||
 	    register_callback(CALLBACKTYPE_failsafe, xen_failsafe_callback))
 		BUG();
 
arch/x86/xen/smp_pv.c +2 -1
@@ -27,6 +27,7 @@
 #include <asm/paravirt.h>
 #include <asm/desc.h>
 #include <asm/pgtable.h>
+#include <asm/idtentry.h>
 #include <asm/cpu.h>
 
 #include <xen/interface/xen.h>
@@ -347,7 +348,7 @@ cpu_initialize_context(unsigned int cpu,
 	ctxt->gs_base_kernel = per_cpu_offset(cpu);
 #endif
 	ctxt->event_callback_eip    =
-		(unsigned long)xen_hypervisor_callback;
+		(unsigned long)xen_asm_exc_xen_hypervisor_callback;
 	ctxt->failsafe_callback_eip =
 		(unsigned long)xen_failsafe_callback;
 	per_cpu(xen_cr3, cpu) = __pa(swapper_pg_dir);
arch/x86/xen/xen-asm_32.S +7 -5
@@ -93,7 +93,7 @@ SYM_CODE_START(xen_iret)
 
 	/*
 	 * If there's something pending, mask events again so we can
-	 * jump back into xen_hypervisor_callback. Otherwise do not
+	 * jump back into exc_xen_hypervisor_callback. Otherwise do not
 	 * touch XEN_vcpu_info_mask.
 	 */
 	jne 1f
@@ -113,7 +113,7 @@ SYM_CODE_START(xen_iret)
 	 * Events are masked, so jumping out of the critical region is
 	 * OK.
 	 */
-	je xen_hypervisor_callback
+	je asm_exc_xen_hypervisor_callback
 
 1:	iret
 xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret)
 	.globl xen_iret_start_crit, xen_iret_end_crit
 
 /*
- * This is called by xen_hypervisor_callback in entry_32.S when it sees
+ * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees
  * that the EIP at the time of interrupt was between
  * xen_iret_start_crit and xen_iret_end_crit.
  *
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret)
  *	 eflags		}
  *	 cs		}  nested exception info
  *	 eip		}
- *	 return address	: (into xen_hypervisor_callback)
+ *	 return address	: (into asm_exc_xen_hypervisor_callback)
  *
  * In order to deliver the nested exception properly, we need to discard the
  * nested exception frame such that when we handle the exception, we do it
@@ -152,7 +152,8 @@ SYM_CODE_END(xen_iret)
  *
  * The only caveat is that if the outer eax hasn't been restored yet (i.e.
  * it's still on stack), we need to restore its value here.
- */
+*/
+.pushsection .noinstr.text, "ax"
 SYM_CODE_START(xen_iret_crit_fixup)
 	/*
 	 * Paranoia: Make sure we're really coming from kernel space.
@@ -181,3 +182,4 @@ SYM_CODE_START(xen_iret_crit_fixup)
 2:
 	ret
 SYM_CODE_END(xen_iret_crit_fixup)
+.popsection
arch/x86/xen/xen-asm_64.S +1 -1
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_err
 #ifdef CONFIG_IA32_EMULATION
 xen_pv_trap entry_INT80_compat
 #endif
-xen_pv_trap hypervisor_callback
+xen_pv_trap asm_exc_xen_hypervisor_callback
 
 	__INIT
 SYM_CODE_START(xen_early_idt_handler_array)
arch/x86/xen/xen-ops.h +0 -1
@@ -8,7 +8,6 @@
 #include <xen/xen-ops.h>
 
 /* These are code, but not functions.  Defined in entry.S */
-extern const char xen_hypervisor_callback[];
 extern const char xen_failsafe_callback[];
 
 void xen_sysenter_target(void);
drivers/xen/Makefile +1 -1
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_HOTPLUG_CPU)		+= cpu_hotplug.o
-obj-y	+= grant-table.o features.o balloon.o manage.o preempt.o time.o
+obj-y	+= grant-table.o features.o balloon.o manage.o time.o
 obj-y	+= mem-reservation.o
 obj-y	+= events/
 obj-y	+= xenbus/
drivers/xen/preempt.c deleted +0 -42

42 lines not on this page · raw message

#48 lore raw

PATCH v9 13/39 x86/entry: Switch XEN/PV hypercall entry to IDTENTRY

Convert the XEN/PV hypercall to IDTENTRY:

- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Remove the old prototypes

The handler stubs need to stay in ASM code as it needs corner case handling and adjustment of the stack pointer.

Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack if required.

The exit code is slightly different from the regular idtentry_exit() on non-preemptible kernels. If the hypercall is preemptible and need_resched() is set then XEN provides a preempt hypercall scheduling function.

Move this functionality into the entry code so it can use the existing idtentry functionality.

---

V9-1: Fixes the fatfingered inline
V9:   Simplified the schedule magic
---
 arch/x86/entry/common.c         |   73 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/entry/entry_32.S       |   17 ++++-----
 arch/x86/entry/entry_64.S       |   22 +++---------
 arch/x86/include/asm/idtentry.h |   13 +++++++
 arch/x86/xen/setup.c            |    4 +-
 arch/x86/xen/smp_pv.c           |    3 +
 arch/x86/xen/xen-asm_32.S       |   12 +++---
 arch/x86/xen/xen-asm_64.S       |    2 -
 arch/x86/xen/xen-ops.h          |    1 
 drivers/xen/Makefile            |    2 -
 drivers/xen/preempt.c           |   42 -----------------------
 11 files changed, 116 insertions(+), 75 deletions(-)
arch/x86/entry/common.c +73 -0
@@ -27,6 +27,9 @@
 #include <linux/syscalls.h>
 #include <linux/uaccess.h>
 
+#include <xen/xen-ops.h>
+#include <xen/events.h>
+
 #include <asm/desc.h>
 #include <asm/traps.h>
 #include <asm/vdso.h>
@@ -35,6 +38,7 @@
 #include <asm/nospec-branch.h>
 #include <asm/io_bitmap.h>
 #include <asm/syscall.h>
+#include <asm/irq_stack.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/syscalls.h>
@@ -677,3 +681,72 @@ void noinstr idtentry_exit_user(struct p
 
 	prepare_exit_to_usermode(regs);
 }
+
+#ifdef CONFIG_XEN_PV
+
+#ifndef CONFIG_PREEMPTION
+/*
+ * Some hypercalls issued by the toolstack can take many 10s of
+ * seconds. Allow tasks running hypercalls via the privcmd driver to
+ * be voluntarily preempted even if full kernel preemption is
+ * disabled.
+ *
+ * Such preemptible hypercalls are bracketed by
+ * xen_preemptible_hcall_begin() and xen_preemptible_hcall_end()
+ * calls.
+ */
+DEFINE_PER_CPU(bool, xen_in_preemptible_hcall);
+EXPORT_SYMBOL_GPL(xen_in_preemptible_hcall);
+
+/*
+ * In case of scheduling the flag must be cleared and restored after
+ * returning from schedule as the task might move to a different CPU.
+ */
+static __always_inline bool get_and_clear_inhcall(void)
+{
+	bool inhcall = __this_cpu_read(xen_in_preemptible_hcall);
+
+	__this_cpu_write(xen_in_preemptible_hcall, false);
+	return inhcall;
+}
+
+static __always_inline void restore_inhcall(bool inhcall)
+{
+	__this_cpu_write(xen_in_preemptible_hcall, inhcall);
+}
+#else
+static __always_inline bool get_and_clear_inhcall(void) { return false; }
+static __always_inline void restore_inhcall(bool inhcall) { }
+#endif
+
+static void __xen_pv_evtchn_do_upcall(void)
+{
+	irq_enter_rcu();
+	inc_irq_stat(irq_hv_callback_count);
+
+	xen_hvm_evtchn_do_upcall();
+
+	irq_exit_rcu();
+}
+
+__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
+{
+	struct pt_regs *old_regs;
+	bool inhcall, rcu_exit;
+
+	rcu_exit = idtentry_enter_cond_rcu(regs);
+	old_regs = set_irq_regs(regs);
+
+	run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, NULL, regs);
+
+	set_irq_regs(old_regs);
+
+	inhcall = get_and_clear_inhcall();
+	if (inhcall && !WARN_ON_ONCE(rcu_exit)) {
+		idtentry_exit_cond_resched(regs, true);
+		restore_inhcall(inhcall);
+	} else {
+		idtentry_exit_cond_rcu(regs, rcu_exit);
+	}
+}
+#endif /* CONFIG_XEN_PV */
arch/x86/entry/entry_32.S +9 -8
@@ -1298,7 +1298,10 @@ SYM_CODE_END(native_iret)
 #endif
 
 #ifdef CONFIG_XEN_PV
-SYM_FUNC_START(xen_hypervisor_callback)
+/*
+ * See comment in entry_64.S for further explanation
+ */
+SYM_FUNC_START(exc_xen_hypervisor_callback)
 	/*
 	 * Check to see if we got the event in the critical
 	 * region in xen_iret_direct, after we've reenabled
@@ -1315,14 +1318,11 @@ SYM_FUNC_START(xen_hypervisor_callback)
 	pushl	$-1				/* orig_ax = -1 => not a system call */
 	SAVE_ALL
 	ENCODE_FRAME_POINTER
-	TRACE_IRQS_OFF
+
 	mov	%esp, %eax
-	call	xen_evtchn_do_upcall
-#ifndef CONFIG_PREEMPTION
-	call	xen_maybe_preempt_hcall
-#endif
-	jmp	ret_from_intr
-SYM_FUNC_END(xen_hypervisor_callback)
+	call	xen_pv_evtchn_do_upcall
+	jmp	handle_exception_return
+SYM_FUNC_END(exc_xen_hypervisor_callback)
 
 /*
  * Hypervisor uses this for application faults while it executes.
@@ -1464,6 +1464,7 @@ SYM_CODE_START_LOCAL_NOALIGN(handle_exce
 	movl	%esp, %eax			# pt_regs pointer
 	CALL_NOSPEC edi
 
+handle_exception_return:
 #ifdef CONFIG_VM86
 	movl	PT_EFLAGS(%esp), %eax		# mix EFLAGS and CS
 	movb	PT_CS(%esp), %al
arch/x86/entry/entry_64.S +7 -15
@@ -1067,10 +1067,6 @@ apicinterrupt IRQ_WORK_VECTOR			irq_work
 
 idtentry	X86_TRAP_PF		page_fault		do_page_fault			has_error_code=1
 
-#ifdef CONFIG_XEN_PV
-idtentry	512 /* dummy */		hypervisor_callback	xen_do_hypervisor_callback	has_error_code=0
-#endif
-
 /*
  * Reload gs selector with exception handling
  * edi:  new selector
@@ -1158,9 +1154,10 @@ SYM_FUNC_END(asm_call_on_stack)
  * So, on entry to the handler we detect whether we interrupted an
  * existing activation in its critical region -- if so, we pop the current
  * activation and restart the handler using the previous one.
+ *
+ * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs)
  */
-/* do_hypervisor_callback(struct *pt_regs) */
-SYM_CODE_START_LOCAL(xen_do_hypervisor_callback)
+SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback)
 
 /*
  * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1170,15 +1167,10 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_c
 	movq	%rdi, %rsp			/* we don't return, adjust the stack frame */
 	UNWIND_HINT_REGS
 
-	ENTER_IRQ_STACK old_rsp=%r10
-	call	xen_evtchn_do_upcall
-	LEAVE_IRQ_STACK
-
-#ifndef CONFIG_PREEMPTION
-	call	xen_maybe_preempt_hcall
-#endif
-	jmp	error_exit
-SYM_CODE_END(xen_do_hypervisor_callback)
+	call	xen_pv_evtchn_do_upcall
+
+	jmp	error_return
+SYM_CODE_END(exc_xen_hypervisor_callback)
 
 /*
  * Hypervisor uses this for application faults while it executes.
arch/x86/include/asm/idtentry.h +13 -0
@@ -337,6 +337,13 @@ static __always_inline void __##func(str
  * This avoids duplicate defines and ensures that everything is consistent.
  */
 
+/*
+ * Dummy trap number so the low level ASM macro vector number checks do not
+ * match which results in emitting plain IDTENTRY stubs without bells and
+ * whistels.
+ */
+#define X86_TRAP_OTHER		0xFFFF
+
 /* Simple exception entry points. No hardware error code */
 DECLARE_IDTENTRY(X86_TRAP_DE,		exc_divide_error);
 DECLARE_IDTENTRY(X86_TRAP_OF,		exc_overflow);
@@ -376,4 +383,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB,	debug)
 /* #DF */
 DECLARE_IDTENTRY_DF(X86_TRAP_DF,	exc_double_fault);
 
+#ifdef CONFIG_XEN_PV
+DECLARE_IDTENTRY(X86_TRAP_OTHER,	exc_xen_hypervisor_callback);
+#endif
+
+#undef X86_TRAP_OTHER
+
 #endif
arch/x86/xen/setup.c +3 -1
@@ -20,6 +20,7 @@
 #include <asm/setup.h>
 #include <asm/acpi.h>
 #include <asm/numa.h>
+#include <asm/idtentry.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
 
@@ -993,7 +994,8 @@ static void __init xen_pvmmu_arch_setup(
 	HYPERVISOR_vm_assist(VMASST_CMD_enable,
 			     VMASST_TYPE_pae_extended_cr3);
 
-	if (register_callback(CALLBACKTYPE_event, xen_hypervisor_callback) ||
+	if (register_callback(CALLBACKTYPE_event,
+			      xen_asm_exc_xen_hypervisor_callback) ||
 	    register_callback(CALLBACKTYPE_failsafe, xen_failsafe_callback))
 		BUG();
 
arch/x86/xen/smp_pv.c +2 -1
@@ -27,6 +27,7 @@
 #include <asm/paravirt.h>
 #include <asm/desc.h>
 #include <asm/pgtable.h>
+#include <asm/idtentry.h>
 #include <asm/cpu.h>
 
 #include <xen/interface/xen.h>
@@ -347,7 +348,7 @@ cpu_initialize_context(unsigned int cpu,
 	ctxt->gs_base_kernel = per_cpu_offset(cpu);
 #endif
 	ctxt->event_callback_eip    =
-		(unsigned long)xen_hypervisor_callback;
+		(unsigned long)xen_asm_exc_xen_hypervisor_callback;
 	ctxt->failsafe_callback_eip =
 		(unsigned long)xen_failsafe_callback;
 	per_cpu(xen_cr3, cpu) = __pa(swapper_pg_dir);
arch/x86/xen/xen-asm_32.S +7 -5
@@ -93,7 +93,7 @@ SYM_CODE_START(xen_iret)
 
 	/*
 	 * If there's something pending, mask events again so we can
-	 * jump back into xen_hypervisor_callback. Otherwise do not
+	 * jump back into exc_xen_hypervisor_callback. Otherwise do not
 	 * touch XEN_vcpu_info_mask.
 	 */
 	jne 1f
@@ -113,7 +113,7 @@ SYM_CODE_START(xen_iret)
 	 * Events are masked, so jumping out of the critical region is
 	 * OK.
 	 */
-	je xen_hypervisor_callback
+	je asm_exc_xen_hypervisor_callback
 
 1:	iret
 xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret)
 	.globl xen_iret_start_crit, xen_iret_end_crit
 
 /*
- * This is called by xen_hypervisor_callback in entry_32.S when it sees
+ * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees
  * that the EIP at the time of interrupt was between
  * xen_iret_start_crit and xen_iret_end_crit.
  *
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret)
  *	 eflags		}
  *	 cs		}  nested exception info
  *	 eip		}
- *	 return address	: (into xen_hypervisor_callback)
+ *	 return address	: (into asm_exc_xen_hypervisor_callback)
  *
  * In order to deliver the nested exception properly, we need to discard the
  * nested exception frame such that when we handle the exception, we do it
@@ -152,7 +152,8 @@ SYM_CODE_END(xen_iret)
  *
  * The only caveat is that if the outer eax hasn't been restored yet (i.e.
  * it's still on stack), we need to restore its value here.
- */
+*/
+.pushsection .noinstr.text, "ax"
 SYM_CODE_START(xen_iret_crit_fixup)
 	/*
 	 * Paranoia: Make sure we're really coming from kernel space.
@@ -181,3 +182,4 @@ SYM_CODE_START(xen_iret_crit_fixup)
 2:
 	ret
 SYM_CODE_END(xen_iret_crit_fixup)
+.popsection
arch/x86/xen/xen-asm_64.S +1 -1
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_err
 #ifdef CONFIG_IA32_EMULATION
 xen_pv_trap entry_INT80_compat
 #endif
-xen_pv_trap hypervisor_callback
+xen_pv_trap asm_exc_xen_hypervisor_callback
 
 	__INIT
 SYM_CODE_START(xen_early_idt_handler_array)
arch/x86/xen/xen-ops.h +0 -1
@@ -8,7 +8,6 @@
 #include <xen/xen-ops.h>
 
 /* These are code, but not functions.  Defined in entry.S */
-extern const char xen_hypervisor_callback[];
 extern const char xen_failsafe_callback[];
 
 void xen_sysenter_target(void);
drivers/xen/Makefile +1 -1
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_HOTPLUG_CPU)		+= cpu_hotplug.o
-obj-y	+= grant-table.o features.o balloon.o manage.o preempt.o time.o
+obj-y	+= grant-table.o features.o balloon.o manage.o time.o
 obj-y	+= mem-reservation.o
 obj-y	+= events/
 obj-y	+= xenbus/
drivers/xen/preempt.c deleted +0 -42

42 lines not on this page · raw message

maintainer Acked-by Tested-by on v9 13/39
#49 lore raw

Re: [patch V9-1 13/39] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY

Quoted 13 lines · Thomas Gleixner

Convert the XEN/PV hypercall to IDTENTRY:

- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Remove the old prototypes

The handler stubs need to stay in ASM code as it needs corner case handling and adjustment of the stack pointer.

Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack if required.

Quoted 8 lines

The exit code is slightly different from the regular idtentry_exit() on non-preemptible kernels. If the hypercall is preemptible and need_resched() is set then XEN provides a preempt hypercall scheduling function.

Move this functionality into the entry code so it can use the existing idtentry functionality.

Juergen

#50 lore raw

[tip: x86/entry] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY

In reply to Thomas Gleixner
The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     66a07b44e7658da3145bb3a8f45256d103e3addc
Gitweb:        https://git.kernel.org/tip/66a07b44e7658da3145bb3a8f45256d103e3addc
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:26 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Switch XEN/PV hypercall entry to IDTENTRY

Convert the XEN/PV hypercall to IDTENTRY:

- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64-bit
- Remove the open coded ASM entry code in 32-bit
- Remove the old prototypes

The handler stubs need to stay in ASM code as they need corner case handling and adjustment of the stack pointer.

Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack if required.

The exit code is slightly different from the regular idtentry_exit() on non-preemptible kernels. If the hypercall is preemptible and need_resched() is set then XEN provides a preempt hypercall scheduling function.

Move this functionality into the entry code so it can use the existing idtentry functionality.

[ mingo: Build fixes. ]

---
 arch/x86/entry/common.c         | 74 ++++++++++++++++++++++++++++++++-
 arch/x86/entry/entry_32.S       | 17 +++----
 arch/x86/entry/entry_64.S       | 20 ++-------
 arch/x86/include/asm/idtentry.h | 13 ++++++-
 arch/x86/xen/setup.c            |  4 +-
 arch/x86/xen/smp_pv.c           |  3 +-
 arch/x86/xen/xen-asm_32.S       | 12 ++---
 arch/x86/xen/xen-asm_64.S       |  2 +-
 arch/x86/xen/xen-ops.h          |  1 +-
 drivers/xen/Makefile            |  2 +-
 drivers/xen/preempt.c           | 42 +------------------
 include/xen/xen-ops.h           | 19 ++------
 12 files changed, 123 insertions(+), 86 deletions(-)
 delete mode 100644 drivers/xen/preempt.c
arch/x86/entry/common.c +74 -0
index 066215a..bb66f4d 100644
@@ -27,6 +27,11 @@
 #include <linux/syscalls.h>
 #include <linux/uaccess.h>
 
+#ifdef CONFIG_XEN_PV
+#include <xen/xen-ops.h>
+#include <xen/events.h>
+#endif
+
 #include <asm/desc.h>
 #include <asm/traps.h>
 #include <asm/vdso.h>
@@ -35,6 +40,7 @@
 #include <asm/nospec-branch.h>
 #include <asm/io_bitmap.h>
 #include <asm/syscall.h>
+#include <asm/irq_stack.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/syscalls.h>
@@ -680,3 +686,71 @@ void noinstr idtentry_exit_user(struct pt_regs *regs)
 
 	prepare_exit_to_usermode(regs);
 }
+
+#ifdef CONFIG_XEN_PV
+#ifndef CONFIG_PREEMPTION
+/*
+ * Some hypercalls issued by the toolstack can take many 10s of
+ * seconds. Allow tasks running hypercalls via the privcmd driver to
+ * be voluntarily preempted even if full kernel preemption is
+ * disabled.
+ *
+ * Such preemptible hypercalls are bracketed by
+ * xen_preemptible_hcall_begin() and xen_preemptible_hcall_end()
+ * calls.
+ */
+DEFINE_PER_CPU(bool, xen_in_preemptible_hcall);
+EXPORT_SYMBOL_GPL(xen_in_preemptible_hcall);
+
+/*
+ * In case of scheduling the flag must be cleared and restored after
+ * returning from schedule as the task might move to a different CPU.
+ */
+static __always_inline bool get_and_clear_inhcall(void)
+{
+	bool inhcall = __this_cpu_read(xen_in_preemptible_hcall);
+
+	__this_cpu_write(xen_in_preemptible_hcall, false);
+	return inhcall;
+}
+
+static __always_inline void restore_inhcall(bool inhcall)
+{
+	__this_cpu_write(xen_in_preemptible_hcall, inhcall);
+}
+#else
+static __always_inline bool get_and_clear_inhcall(void) { return false; }
+static __always_inline void restore_inhcall(bool inhcall) { }
+#endif
+
+static void __xen_pv_evtchn_do_upcall(void)
+{
+	irq_enter_rcu();
+	inc_irq_stat(irq_hv_callback_count);
+
+	xen_hvm_evtchn_do_upcall();
+
+	irq_exit_rcu();
+}
+
+__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)
+{
+	struct pt_regs *old_regs;
+	bool inhcall, rcu_exit;
+
+	rcu_exit = idtentry_enter_cond_rcu(regs);
+	old_regs = set_irq_regs(regs);
+
+	run_on_irqstack_cond(__xen_pv_evtchn_do_upcall, NULL, regs);
+
+	set_irq_regs(old_regs);
+
+	inhcall = get_and_clear_inhcall();
+	if (inhcall && !WARN_ON_ONCE(rcu_exit)) {
+		idtentry_exit_cond_resched(regs, true);
+		restore_inhcall(inhcall);
+	} else {
+		idtentry_exit_cond_rcu(regs, rcu_exit);
+	}
+}
+#endif /* CONFIG_XEN_PV */
arch/x86/entry/entry_32.S +9 -8
index 28d13f0..84ee014 100644
@@ -1298,7 +1298,10 @@ SYM_CODE_END(native_iret)
 #endif
 
 #ifdef CONFIG_XEN_PV
-SYM_FUNC_START(xen_hypervisor_callback)
+/*
+ * See comment in entry_64.S for further explanation
+ */
+SYM_FUNC_START(exc_xen_hypervisor_callback)
 	/*
 	 * Check to see if we got the event in the critical
 	 * region in xen_iret_direct, after we've reenabled
@@ -1315,14 +1318,11 @@ SYM_FUNC_START(xen_hypervisor_callback)
 	pushl	$-1				/* orig_ax = -1 => not a system call */
 	SAVE_ALL
 	ENCODE_FRAME_POINTER
-	TRACE_IRQS_OFF
+
 	mov	%esp, %eax
-	call	xen_evtchn_do_upcall
-#ifndef CONFIG_PREEMPTION
-	call	xen_maybe_preempt_hcall
-#endif
-	jmp	ret_from_intr
-SYM_FUNC_END(xen_hypervisor_callback)
+	call	xen_pv_evtchn_do_upcall
+	jmp	handle_exception_return
+SYM_FUNC_END(exc_xen_hypervisor_callback)
 
 /*
  * Hypervisor uses this for application faults while it executes.
@@ -1464,6 +1464,7 @@ SYM_CODE_START_LOCAL_NOALIGN(handle_exception)
 	movl	%esp, %eax			# pt_regs pointer
 	CALL_NOSPEC edi
 
+handle_exception_return:
 #ifdef CONFIG_VM86
 	movl	PT_EFLAGS(%esp), %eax		# mix EFLAGS and CS
 	movb	PT_CS(%esp), %al
arch/x86/entry/entry_64.S +6 -14
index 6b518be..dadb37d 100644
@@ -1067,10 +1067,6 @@ apicinterrupt IRQ_WORK_VECTOR			irq_work_interrupt		smp_irq_work_interrupt
 
 idtentry	X86_TRAP_PF		page_fault		do_page_fault			has_error_code=1
 
-#ifdef CONFIG_XEN_PV
-idtentry	512 /* dummy */		hypervisor_callback	xen_do_hypervisor_callback	has_error_code=0
-#endif
-
 /*
  * Reload gs selector with exception handling
  * edi:  new selector
@@ -1158,9 +1154,10 @@ SYM_FUNC_END(asm_call_on_stack)
  * So, on entry to the handler we detect whether we interrupted an
  * existing activation in its critical region -- if so, we pop the current
  * activation and restart the handler using the previous one.
+ *
+ * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs)
  */
-/* do_hypervisor_callback(struct *pt_regs) */
-SYM_CODE_START_LOCAL(xen_do_hypervisor_callback)
+SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback)
 
 /*
  * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1170,15 +1167,10 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_callback)
 	movq	%rdi, %rsp			/* we don't return, adjust the stack frame */
 	UNWIND_HINT_REGS
 
-	ENTER_IRQ_STACK old_rsp=%r10
-	call	xen_evtchn_do_upcall
-	LEAVE_IRQ_STACK
+	call	xen_pv_evtchn_do_upcall
 
-#ifndef CONFIG_PREEMPTION
-	call	xen_maybe_preempt_hcall
-#endif
-	jmp	error_exit
-SYM_CODE_END(xen_do_hypervisor_callback)
+	jmp	error_return
+SYM_CODE_END(exc_xen_hypervisor_callback)
 
 /*
  * Hypervisor uses this for application faults while it executes.
arch/x86/include/asm/idtentry.h +13 -0
index b056889..49a797d 100644
@@ -337,6 +337,13 @@ __visible noinstr void func(struct pt_regs *regs,			\
  * This avoids duplicate defines and ensures that everything is consistent.
  */
 
+/*
+ * Dummy trap number so the low level ASM macro vector number checks do not
+ * match which results in emitting plain IDTENTRY stubs without bells and
+ * whistels.
+ */
+#define X86_TRAP_OTHER		0xFFFF
+
 /* Simple exception entry points. No hardware error code */
 DECLARE_IDTENTRY(X86_TRAP_DE,		exc_divide_error);
 DECLARE_IDTENTRY(X86_TRAP_OF,		exc_overflow);
@@ -376,4 +383,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB,	debug);
 /* #DF */
 DECLARE_IDTENTRY_DF(X86_TRAP_DF,	exc_double_fault);
 
+#ifdef CONFIG_XEN_PV
+DECLARE_IDTENTRY(X86_TRAP_OTHER,	exc_xen_hypervisor_callback);
+#endif
+
+#undef X86_TRAP_OTHER
+
 #endif
arch/x86/xen/setup.c +3 -1
index 1a2d8a5..3566e37 100644
@@ -20,6 +20,7 @@
 #include <asm/setup.h>
 #include <asm/acpi.h>
 #include <asm/numa.h>
+#include <asm/idtentry.h>
 #include <asm/xen/hypervisor.h>
 #include <asm/xen/hypercall.h>
 
@@ -993,7 +994,8 @@ static void __init xen_pvmmu_arch_setup(void)
 	HYPERVISOR_vm_assist(VMASST_CMD_enable,
 			     VMASST_TYPE_pae_extended_cr3);
 
-	if (register_callback(CALLBACKTYPE_event, xen_hypervisor_callback) ||
+	if (register_callback(CALLBACKTYPE_event,
+			      xen_asm_exc_xen_hypervisor_callback) ||
 	    register_callback(CALLBACKTYPE_failsafe, xen_failsafe_callback))
 		BUG();
 
arch/x86/xen/smp_pv.c +2 -1
index 8fb8a50..a92259d 100644
@@ -27,6 +27,7 @@
 #include <asm/paravirt.h>
 #include <asm/desc.h>
 #include <asm/pgtable.h>
+#include <asm/idtentry.h>
 #include <asm/cpu.h>
 
 #include <xen/interface/xen.h>
@@ -347,7 +348,7 @@ cpu_initialize_context(unsigned int cpu, struct task_struct *idle)
 	ctxt->gs_base_kernel = per_cpu_offset(cpu);
 #endif
 	ctxt->event_callback_eip    =
-		(unsigned long)xen_hypervisor_callback;
+		(unsigned long)xen_asm_exc_xen_hypervisor_callback;
 	ctxt->failsafe_callback_eip =
 		(unsigned long)xen_failsafe_callback;
 	per_cpu(xen_cr3, cpu) = __pa(swapper_pg_dir);
arch/x86/xen/xen-asm_32.S +7 -5
index 812ff01..d0ff2dc 100644
@@ -93,7 +93,7 @@ xen_iret_start_crit:
 
 	/*
 	 * If there's something pending, mask events again so we can
-	 * jump back into xen_hypervisor_callback. Otherwise do not
+	 * jump back into exc_xen_hypervisor_callback. Otherwise do not
 	 * touch XEN_vcpu_info_mask.
 	 */
 	jne 1f
@@ -113,7 +113,7 @@ iret_restore_end:
 	 * Events are masked, so jumping out of the critical region is
 	 * OK.
 	 */
-	je xen_hypervisor_callback
+	je asm_exc_xen_hypervisor_callback
 
 1:	iret
 xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret)
 	.globl xen_iret_start_crit, xen_iret_end_crit
 
 /*
- * This is called by xen_hypervisor_callback in entry_32.S when it sees
+ * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees
  * that the EIP at the time of interrupt was between
  * xen_iret_start_crit and xen_iret_end_crit.
  *
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret)
  *	 eflags		}
  *	 cs		}  nested exception info
  *	 eip		}
- *	 return address	: (into xen_hypervisor_callback)
+ *	 return address	: (into asm_exc_xen_hypervisor_callback)
  *
  * In order to deliver the nested exception properly, we need to discard the
  * nested exception frame such that when we handle the exception, we do it
@@ -152,7 +152,8 @@ SYM_CODE_END(xen_iret)
  *
  * The only caveat is that if the outer eax hasn't been restored yet (i.e.
  * it's still on stack), we need to restore its value here.
- */
+*/
+.pushsection .noinstr.text, "ax"
 SYM_CODE_START(xen_iret_crit_fixup)
 	/*
 	 * Paranoia: Make sure we're really coming from kernel space.
@@ -181,3 +182,4 @@ SYM_CODE_START(xen_iret_crit_fixup)
 2:
 	ret
 SYM_CODE_END(xen_iret_crit_fixup)
+.popsection
arch/x86/xen/xen-asm_64.S +1 -1
index e46d863..19fbbdb 100644
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_error
 #ifdef CONFIG_IA32_EMULATION
 xen_pv_trap entry_INT80_compat
 #endif
-xen_pv_trap hypervisor_callback
+xen_pv_trap asm_exc_xen_hypervisor_callback
 
 	__INIT
 SYM_CODE_START(xen_early_idt_handler_array)
arch/x86/xen/xen-ops.h +0 -1
index 1cc1568..ad05d05 100644
@@ -8,7 +8,6 @@
 #include <xen/xen-ops.h>
 
 /* These are code, but not functions.  Defined in entry.S */
-extern const char xen_hypervisor_callback[];
 extern const char xen_failsafe_callback[];
 
 void xen_sysenter_target(void);
drivers/xen/Makefile +1 -1
index 0c4efa6..0d322f3 100644
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_HOTPLUG_CPU)		+= cpu_hotplug.o
-obj-y	+= grant-table.o features.o balloon.o manage.o preempt.o time.o
+obj-y	+= grant-table.o features.o balloon.o manage.o time.o
 obj-y	+= mem-reservation.o
 obj-y	+= events/
 obj-y	+= xenbus/
drivers/xen/preempt.c deleted +0 -42

42 lines not on this page · raw message

include/xen/xen-ops.h +7 -12
index 095be1d..39a5580 100644
@@ -215,17 +215,7 @@ bool xen_running_on_version_or_later(unsigned int major, unsigned int minor);
 void xen_efi_runtime_setup(void);
 
 
-#ifdef CONFIG_PREEMPTION
-
-static inline void xen_preemptible_hcall_begin(void)
-{
-}
-
-static inline void xen_preemptible_hcall_end(void)
-{
-}
-
-#else
+#if defined(CONFIG_XEN_PV) && !defined(CONFIG_PREEMPTION)
 
 DECLARE_PER_CPU(bool, xen_in_preemptible_hcall);
 
@@ -239,6 +229,11 @@ static inline void xen_preemptible_hcall_end(void)
 	__this_cpu_write(xen_in_preemptible_hcall, false);
 }
 
-#endif /* CONFIG_PREEMPTION */
+#else
+
+static inline void xen_preemptible_hcall_begin(void) { }
+static inline void xen_preemptible_hcall_end(void) { }
+
+#endif /* CONFIG_XEN_PV && !CONFIG_PREEMPTION */
 
 #endif /* INCLUDE_XEN_OPS_H */
#51 lore raw

PATCH v9 14/39 x86/entry/64: Simplify idtentry_body

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

All C functions which do not have an error code have been converted to the new IDTENTRY interface which does not expect an error code in the arguments. Spare the XORL.

---
 arch/x86/entry/entry_64.S |    2 --
 1 file changed, 2 deletions(-)
arch/x86/entry/entry_64.S +0 -2
@@ -531,8 +531,6 @@ SYM_CODE_END(spurious_entries_start)
 	.if \has_error_code == 1
 		movq	ORIG_RAX(%rsp), %rsi	/* get error code into 2nd argument*/
 		movq	$-1, ORIG_RAX(%rsp)	/* no syscall to restart */
-	.else
-		xorl	%esi, %esi		/* Clear the error code */
 	.endif
 
 	.if \vector == X86_TRAP_PF
#52 lore raw

[tip: x86/entry] x86/entry/64: Simplify idtentry_body

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     8518ad5d38056deee232b186533a18e7722ad3ad
Gitweb:        https://git.kernel.org/tip/8518ad5d38056deee232b186533a18e7722ad3ad
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:27 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry/64: Simplify idtentry_body

All C functions which do not have an error code have been converted to the new IDTENTRY interface which does not expect an error code in the arguments. Spare the XORL.

---
 arch/x86/entry/entry_64.S | 2 --
 1 file changed, 2 deletions(-)
arch/x86/entry/entry_64.S +0 -2
index dadb37d..b70c778 100644
@@ -531,8 +531,6 @@ SYM_CODE_END(spurious_entries_start)
 	.if \has_error_code == 1
 		movq	ORIG_RAX(%rsp), %rsi	/* get error code into 2nd argument*/
 		movq	$-1, ORIG_RAX(%rsp)	/* no syscall to restart */
-	.else
-		xorl	%esi, %esi		/* Clear the error code */
 	.endif
 
 	.if \vector == X86_TRAP_PF
#53 lore raw

PATCH v9 15/39 x86/entry: Switch page fault exception to IDTENTRY_RAW

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Convert page fault exceptions to IDTENTRY_RAW:
- Implement the C entry point with DEFINE_IDTENTRY_RAW
- Add the CR2 read into the exception handler
- Add the idtentry_enter/exit_cond_rcu() invocations in in the regular page fault handler and in the async PF part.
- Emit the ASM stub with DECLARE_IDTENTRY_RAW
- Remove the ASM idtentry in 64bit
- Remove the CR2 read from 64bit
- Remove the open coded ASM entry code in 32bit
- Fixup the XEN/PV code
- Remove the old prototypes

No functional change.

---
V9: Adopt async #PF to conditional RCU handling

---
 arch/x86/entry/entry_32.S       |   30 -----------------
 arch/x86/entry/entry_64.S       |   19 -----------
 arch/x86/include/asm/idtentry.h |    3 +
 arch/x86/include/asm/traps.h    |   11 ------
 arch/x86/kernel/idt.c           |    4 +-
 arch/x86/kernel/kvm.c           |   15 +++++---
 arch/x86/mm/fault.c             |   69 +++++++++++++++++++++++++++-------------
 arch/x86/xen/enlighten_pv.c     |    2 -
 arch/x86/xen/xen-asm_64.S       |    2 -
 9 files changed, 63 insertions(+), 92 deletions(-)
arch/x86/entry/entry_32.S +0 -30
@@ -1395,36 +1395,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vec
 
 #endif /* CONFIG_HYPERV */
 
-SYM_CODE_START(page_fault)
-	ASM_CLAC
-	pushl	$do_page_fault
-	jmp	common_exception_read_cr2
-SYM_CODE_END(page_fault)
-
-SYM_CODE_START_LOCAL_NOALIGN(common_exception_read_cr2)
-	/* the function address is in %gs's slot on the stack */
-	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
-
-	ENCODE_FRAME_POINTER
-
-	/* fixup %gs */
-	GS_TO_REG %ecx
-	movl	PT_GS(%esp), %edi
-	REG_TO_PTGS %ecx
-	SET_KERNEL_GS %ecx
-
-	GET_CR2_INTO(%ecx)			# might clobber %eax
-
-	/* fixup orig %eax */
-	movl	PT_ORIG_EAX(%esp), %edx		# get the error code
-	movl	$-1, PT_ORIG_EAX(%esp)		# no syscall to restart
-
-	TRACE_IRQS_OFF
-	movl	%esp, %eax			# pt_regs pointer
-	CALL_NOSPEC edi
-	jmp	ret_from_exception
-SYM_CODE_END(common_exception_read_cr2)
-
 SYM_CODE_START_LOCAL_NOALIGN(common_exception)
 	/* the function address is in %gs's slot on the stack */
 	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -19
@@ -506,15 +506,6 @@ SYM_CODE_END(spurious_entries_start)
 	call	error_entry
 	UNWIND_HINT_REGS
 
-	.if \vector == X86_TRAP_PF
-		/*
-		 * Store CR2 early so subsequent faults cannot clobber it. Use R12 as
-		 * intermediate storage as RDX can be clobbered in enter_from_user_mode().
-		 * GET_CR2_INTO can clobber RAX.
-		 */
-		GET_CR2_INTO(%r12);
-	.endif
-
 	.if \sane == 0
 	TRACE_IRQS_OFF
 
@@ -533,10 +524,6 @@ SYM_CODE_END(spurious_entries_start)
 		movq	$-1, ORIG_RAX(%rsp)	/* no syscall to restart */
 	.endif
 
-	.if \vector == X86_TRAP_PF
-		movq	%r12, %rdx		/* Move CR2 into 3rd argument */
-	.endif
-
 	call	\cfunc
 
 	.if \sane == 0
@@ -1060,12 +1047,6 @@ apicinterrupt IRQ_WORK_VECTOR			irq_work
 #endif
 
 /*
- * Exception entry points.
- */
-
-idtentry	X86_TRAP_PF		page_fault		do_page_fault			has_error_code=1
-
-/*
  * Reload gs selector with exception handling
  * edi:  new selector
  *
arch/x86/include/asm/idtentry.h +2 -1
@@ -366,7 +366,8 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_GP,
 DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_AC,	exc_alignment_check);
 
 /* Raw exception entries which need extra work */
-DECLARE_IDTENTRY_RAW(X86_TRAP_BP,	exc_int3);
+DECLARE_IDTENTRY_RAW(X86_TRAP_BP,		exc_int3);
+DECLARE_IDTENTRY_RAW_ERRORCODE(X86_TRAP_PF,	exc_page_fault);
 
 #ifdef CONFIG_X86_MCE
 DECLARE_IDTENTRY_MCE(X86_TRAP_MC,	exc_machine_check);
arch/x86/include/asm/traps.h +0 -11
@@ -9,17 +9,6 @@
 #include <asm/idtentry.h>
 #include <asm/siginfo.h>			/* TRAP_TRACE, ... */
 
-#define dotraplinkage __visible
-
-asmlinkage void page_fault(void);
-asmlinkage void async_page_fault(void);
-
-#if defined(CONFIG_X86_64) && defined(CONFIG_XEN_PV)
-asmlinkage void xen_page_fault(void);
-#endif
-
-dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address);
-
 #ifdef CONFIG_X86_64
 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
 asmlinkage __visible notrace
arch/x86/kernel/idt.c +2 -2
@@ -62,7 +62,7 @@ static const __initconst struct idt_data
 	INTG(X86_TRAP_DB,		asm_exc_debug),
 	SYSG(X86_TRAP_BP,		asm_exc_int3),
 #ifdef CONFIG_X86_32
-	INTG(X86_TRAP_PF,		page_fault),
+	INTG(X86_TRAP_PF,		asm_exc_page_fault),
 #endif
 };
 
@@ -156,7 +156,7 @@ static const __initconst struct idt_data
  * stacks work only after cpu_init().
  */
 static const __initconst struct idt_data early_pf_idts[] = {
-	INTG(X86_TRAP_PF,		page_fault),
+	INTG(X86_TRAP_PF,		asm_exc_page_fault),
 };
 
 /*
arch/x86/kernel/kvm.c +9 -6
@@ -218,7 +218,7 @@ void kvm_async_pf_task_wake(u32 token)
 }
 EXPORT_SYMBOL_GPL(kvm_async_pf_task_wake);
 
-u32 kvm_read_and_reset_pf_reason(void)
+u32 noinstr kvm_read_and_reset_pf_reason(void)
 {
 	u32 reason = 0;
 
@@ -230,11 +230,11 @@ u32 kvm_read_and_reset_pf_reason(void)
 	return reason;
 }
 EXPORT_SYMBOL_GPL(kvm_read_and_reset_pf_reason);
-NOKPROBE_SYMBOL(kvm_read_and_reset_pf_reason);
 
-bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)
+noinstr bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)
 {
 	u32 reason = kvm_read_and_reset_pf_reason();
+	bool rcu_exit;
 
 	switch (reason) {
 	case KVM_PV_REASON_PAGE_NOT_PRESENT:
@@ -244,6 +244,9 @@ bool __kvm_handle_async_pf(struct pt_reg
 		return false;
 	}
 
+	rcu_exit = idtentry_enter_cond_rcu(regs);
+	instrumentation_begin();
+
 	/*
 	 * If the host managed to inject an async #PF into an interrupt
 	 * disabled region, then die hard as this is not going to end well
@@ -258,13 +261,13 @@ bool __kvm_handle_async_pf(struct pt_reg
 		/* Page is swapped out by the host. */
 		kvm_async_pf_task_wait_schedule(token);
 	} else {
-		rcu_irq_enter();
 		kvm_async_pf_task_wake(token);
-		rcu_irq_exit();
 	}
+
+	instrumentation_end();
+	idtentry_exit_cond_rcu(regs, rcu_exit);
 	return true;
 }
-NOKPROBE_SYMBOL(__kvm_handle_async_pf);
 
 static void __init paravirt_ops_setup(void)
 {
arch/x86/mm/fault.c +48 -21

87 lines not on this page · raw message

arch/x86/xen/enlighten_pv.c +1 -1
@@ -627,7 +627,7 @@ static struct trap_array_entry trap_arra
 #ifdef CONFIG_IA32_EMULATION
 	{ entry_INT80_compat,          xen_entry_INT80_compat,          false },
 #endif
-	{ page_fault,                  xen_page_fault,                  false },
+	TRAP_ENTRY(exc_page_fault,			false ),
 	TRAP_ENTRY(exc_divide_error,			false ),
 	TRAP_ENTRY(exc_bounds,				false ),
 	TRAP_ENTRY(exc_invalid_op,			false ),
arch/x86/xen/xen-asm_64.S +1 -1
@@ -43,7 +43,7 @@ xen_pv_trap asm_exc_invalid_tss
 xen_pv_trap asm_exc_segment_not_present
 xen_pv_trap asm_exc_stack_segment
 xen_pv_trap asm_exc_general_protection
-xen_pv_trap page_fault
+xen_pv_trap asm_exc_page_fault
 xen_pv_trap asm_exc_spurious_interrupt_bug
 xen_pv_trap asm_exc_coprocessor_error
 xen_pv_trap asm_exc_alignment_check
#54 lore raw

[tip: x86/entry] x86/entry: Switch page fault exception to IDTENTRY_RAW

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     a707ae1a9bbb3f8649004d4629f4299ffdcecc2a
Gitweb:        https://git.kernel.org/tip/a707ae1a9bbb3f8649004d4629f4299ffdcecc2a
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:28 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Switch page fault exception to IDTENTRY_RAW

Convert page fault exceptions to IDTENTRY_RAW:

- Implement the C entry point with DEFINE_IDTENTRY_RAW
- Add the CR2 read into the exception handler
- Add the idtentry_enter/exit_cond_rcu() invocations in in the regular page fault handler and in the async PF part.
- Emit the ASM stub with DECLARE_IDTENTRY_RAW
- Remove the ASM idtentry in 64-bit
- Remove the CR2 read from 64-bit
- Remove the open coded ASM entry code in 32-bit
- Fix up the XEN/PV code
- Remove the old prototypes

No functional change.

---
 arch/x86/entry/entry_32.S       | 30 +--------------
 arch/x86/entry/entry_64.S       | 19 +---------
 arch/x86/include/asm/idtentry.h |  3 +-
 arch/x86/include/asm/traps.h    | 11 +-----
 arch/x86/kernel/idt.c           |  4 +-
 arch/x86/kernel/kvm.c           | 15 ++++---
 arch/x86/mm/fault.c             | 69 ++++++++++++++++++++++----------
 arch/x86/xen/enlighten_pv.c     |  2 +-
 arch/x86/xen/xen-asm_64.S       |  2 +-
 9 files changed, 63 insertions(+), 92 deletions(-)
arch/x86/entry/entry_32.S +0 -30
index 84ee014..1674deb 100644
@@ -1395,36 +1395,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR,
 
 #endif /* CONFIG_HYPERV */
 
-SYM_CODE_START(page_fault)
-	ASM_CLAC
-	pushl	$do_page_fault
-	jmp	common_exception_read_cr2
-SYM_CODE_END(page_fault)
-
-SYM_CODE_START_LOCAL_NOALIGN(common_exception_read_cr2)
-	/* the function address is in %gs's slot on the stack */
-	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
-
-	ENCODE_FRAME_POINTER
-
-	/* fixup %gs */
-	GS_TO_REG %ecx
-	movl	PT_GS(%esp), %edi
-	REG_TO_PTGS %ecx
-	SET_KERNEL_GS %ecx
-
-	GET_CR2_INTO(%ecx)			# might clobber %eax
-
-	/* fixup orig %eax */
-	movl	PT_ORIG_EAX(%esp), %edx		# get the error code
-	movl	$-1, PT_ORIG_EAX(%esp)		# no syscall to restart
-
-	TRACE_IRQS_OFF
-	movl	%esp, %eax			# pt_regs pointer
-	CALL_NOSPEC edi
-	jmp	ret_from_exception
-SYM_CODE_END(common_exception_read_cr2)
-
 SYM_CODE_START_LOCAL_NOALIGN(common_exception)
 	/* the function address is in %gs's slot on the stack */
 	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -19
index b70c778..5789f76 100644
@@ -506,15 +506,6 @@ SYM_CODE_END(spurious_entries_start)
 	call	error_entry
 	UNWIND_HINT_REGS
 
-	.if \vector == X86_TRAP_PF
-		/*
-		 * Store CR2 early so subsequent faults cannot clobber it. Use R12 as
-		 * intermediate storage as RDX can be clobbered in enter_from_user_mode().
-		 * GET_CR2_INTO can clobber RAX.
-		 */
-		GET_CR2_INTO(%r12);
-	.endif
-
 	.if \sane == 0
 	TRACE_IRQS_OFF
 
@@ -533,10 +524,6 @@ SYM_CODE_END(spurious_entries_start)
 		movq	$-1, ORIG_RAX(%rsp)	/* no syscall to restart */
 	.endif
 
-	.if \vector == X86_TRAP_PF
-		movq	%r12, %rdx		/* Move CR2 into 3rd argument */
-	.endif
-
 	call	\cfunc
 
 	.if \sane == 0
@@ -1060,12 +1047,6 @@ apicinterrupt IRQ_WORK_VECTOR			irq_work_interrupt		smp_irq_work_interrupt
 #endif
 
 /*
- * Exception entry points.
- */
-
-idtentry	X86_TRAP_PF		page_fault		do_page_fault			has_error_code=1
-
-/*
  * Reload gs selector with exception handling
  * edi:  new selector
  *
arch/x86/include/asm/idtentry.h +2 -1
index 49a797d..92054ff 100644
@@ -366,7 +366,8 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_GP,	exc_general_protection);
 DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_AC,	exc_alignment_check);
 
 /* Raw exception entries which need extra work */
-DECLARE_IDTENTRY_RAW(X86_TRAP_BP,	exc_int3);
+DECLARE_IDTENTRY_RAW(X86_TRAP_BP,		exc_int3);
+DECLARE_IDTENTRY_RAW_ERRORCODE(X86_TRAP_PF,	exc_page_fault);
 
 #ifdef CONFIG_X86_MCE
 DECLARE_IDTENTRY_MCE(X86_TRAP_MC,	exc_machine_check);
arch/x86/include/asm/traps.h +0 -11
index f5a2e43..d7de360 100644
@@ -9,17 +9,6 @@
 #include <asm/idtentry.h>
 #include <asm/siginfo.h>			/* TRAP_TRACE, ... */
 
-#define dotraplinkage __visible
-
-asmlinkage void page_fault(void);
-asmlinkage void async_page_fault(void);
-
-#if defined(CONFIG_X86_64) && defined(CONFIG_XEN_PV)
-asmlinkage void xen_page_fault(void);
-#endif
-
-dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address);
-
 #ifdef CONFIG_X86_64
 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs);
 asmlinkage __visible notrace
arch/x86/kernel/idt.c +2 -2
index ec55479..ddb1115 100644
@@ -62,7 +62,7 @@ static const __initconst struct idt_data early_idts[] = {
 	INTG(X86_TRAP_DB,		asm_exc_debug),
 	SYSG(X86_TRAP_BP,		asm_exc_int3),
 #ifdef CONFIG_X86_32
-	INTG(X86_TRAP_PF,		page_fault),
+	INTG(X86_TRAP_PF,		asm_exc_page_fault),
 #endif
 };
 
@@ -156,7 +156,7 @@ static const __initconst struct idt_data apic_idts[] = {
  * stacks work only after cpu_init().
  */
 static const __initconst struct idt_data early_pf_idts[] = {
-	INTG(X86_TRAP_PF,		page_fault),
+	INTG(X86_TRAP_PF,		asm_exc_page_fault),
 };
 
 /*
arch/x86/kernel/kvm.c +9 -6
index b3d9b0d..d8a0926 100644
@@ -218,7 +218,7 @@ again:
 }
 EXPORT_SYMBOL_GPL(kvm_async_pf_task_wake);
 
-u32 kvm_read_and_reset_pf_reason(void)
+u32 noinstr kvm_read_and_reset_pf_reason(void)
 {
 	u32 reason = 0;
 
@@ -230,11 +230,11 @@ u32 kvm_read_and_reset_pf_reason(void)
 	return reason;
 }
 EXPORT_SYMBOL_GPL(kvm_read_and_reset_pf_reason);
-NOKPROBE_SYMBOL(kvm_read_and_reset_pf_reason);
 
-bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)
+noinstr bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)
 {
 	u32 reason = kvm_read_and_reset_pf_reason();
+	bool rcu_exit;
 
 	switch (reason) {
 	case KVM_PV_REASON_PAGE_NOT_PRESENT:
@@ -244,6 +244,9 @@ bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)
 		return false;
 	}
 
+	rcu_exit = idtentry_enter_cond_rcu(regs);
+	instrumentation_begin();
+
 	/*
 	 * If the host managed to inject an async #PF into an interrupt
 	 * disabled region, then die hard as this is not going to end well
@@ -258,13 +261,13 @@ bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)
 		/* Page is swapped out by the host. */
 		kvm_async_pf_task_wait_schedule(token);
 	} else {
-		rcu_irq_enter();
 		kvm_async_pf_task_wake(token);
-		rcu_irq_exit();
 	}
+
+	instrumentation_end();
+	idtentry_exit_cond_rcu(regs, rcu_exit);
 	return true;
 }
-NOKPROBE_SYMBOL(__kvm_handle_async_pf);
 
 static void __init paravirt_ops_setup(void)
 {
arch/x86/mm/fault.c +48 -21

87 lines not on this page · raw message

arch/x86/xen/enlighten_pv.c +1 -1
index 851ea41..35321f4 100644
@@ -627,7 +627,7 @@ static struct trap_array_entry trap_array[] = {
 #ifdef CONFIG_IA32_EMULATION
 	{ entry_INT80_compat,          xen_entry_INT80_compat,          false },
 #endif
-	{ page_fault,                  xen_page_fault,                  false },
+	TRAP_ENTRY(exc_page_fault,			false ),
 	TRAP_ENTRY(exc_divide_error,			false ),
 	TRAP_ENTRY(exc_bounds,				false ),
 	TRAP_ENTRY(exc_invalid_op,			false ),
arch/x86/xen/xen-asm_64.S +1 -1
index 19fbbdb..5d252aa 100644
@@ -43,7 +43,7 @@ xen_pv_trap asm_exc_invalid_tss
 xen_pv_trap asm_exc_segment_not_present
 xen_pv_trap asm_exc_stack_segment
 xen_pv_trap asm_exc_general_protection
-xen_pv_trap page_fault
+xen_pv_trap asm_exc_page_fault
 xen_pv_trap asm_exc_spurious_interrupt_bug
 xen_pv_trap asm_exc_coprocessor_error
 xen_pv_trap asm_exc_alignment_check
#55 lore raw

PATCH v9 16/39 x86/entry: Remove the transition leftovers

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Now that all exceptions are converted over the sane flag is not longer needed. Also the vector argument of idtentry_body on 64 bit is pointless now.

---
 arch/x86/entry/entry_32.S       |    3 +--
 arch/x86/entry/entry_64.S       |   26 ++++----------------------
 arch/x86/include/asm/idtentry.h |    6 +++---
 3 files changed, 8 insertions(+), 27 deletions(-)
arch/x86/entry/entry_32.S +1 -2
@@ -734,9 +734,8 @@
  * @asmsym:		ASM symbol for the entry point
  * @cfunc:		C function to be called
  * @has_error_code:	Hardware pushed error code on stack
- * @sane:		Compatibility flag with 64bit
  */
-.macro idtentry vector asmsym cfunc has_error_code:req sane=0
+.macro idtentry vector asmsym cfunc has_error_code:req
 SYM_CODE_START(\asmsym)
 	ASM_CLAC
 	cld
arch/x86/entry/entry_64.S +4 -22
@@ -496,27 +496,14 @@ SYM_CODE_END(spurious_entries_start)
 
 /**
  * idtentry_body - Macro to emit code calling the C function
- * @vector:		Vector number
  * @cfunc:		C function to be called
  * @has_error_code:	Hardware pushed error code on stack
- * @sane:		Sane variant which handles irq tracing, context tracking in C
  */
-.macro idtentry_body vector cfunc has_error_code:req sane=0
+.macro idtentry_body cfunc has_error_code:req
 
 	call	error_entry
 	UNWIND_HINT_REGS
 
-	.if \sane == 0
-	TRACE_IRQS_OFF
-
-#ifdef CONFIG_CONTEXT_TRACKING
-	testb	$3, CS(%rsp)
-	jz	.Lfrom_kernel_no_ctxt_tracking_\@
-	CALL_enter_from_user_mode
-.Lfrom_kernel_no_ctxt_tracking_\@:
-#endif
-	.endif
-
 	movq	%rsp, %rdi			/* pt_regs pointer into 1st argument*/
 
 	.if \has_error_code == 1
@@ -526,11 +513,7 @@ SYM_CODE_END(spurious_entries_start)
 
 	call	\cfunc
 
-	.if \sane == 0
-	jmp	error_exit
-	.else
 	jmp	error_return
-	.endif
 .endm
 
 /**
@@ -539,12 +522,11 @@ SYM_CODE_END(spurious_entries_start)
  * @asmsym:		ASM symbol for the entry point
  * @cfunc:		C function to be called
  * @has_error_code:	Hardware pushed error code on stack
- * @sane:		Sane variant which handles irq tracing, context tracking in C
  *
  * The macro emits code to set up the kernel context for straight forward
  * and simple IDT entries. No IST stack, no paranoid entry checks.
  */
-.macro idtentry vector asmsym cfunc has_error_code:req sane=0
+.macro idtentry vector asmsym cfunc has_error_code:req
 SYM_CODE_START(\asmsym)
 	UNWIND_HINT_IRET_REGS offset=\has_error_code*8
 	ASM_CLAC
@@ -567,7 +549,7 @@ SYM_CODE_START(\asmsym)
 .Lfrom_usermode_no_gap_\@:
 	.endif
 
-	idtentry_body \vector \cfunc \has_error_code \sane
+	idtentry_body \cfunc \has_error_code
 
 _ASM_NOKPROBE(\asmsym)
 SYM_CODE_END(\asmsym)
@@ -642,7 +624,7 @@ SYM_CODE_START(\asmsym)
 
 	/* Switch to the regular task stack and use the noist entry point */
 .Lfrom_usermode_switch_stack_\@:
-	idtentry_body vector noist_\cfunc, has_error_code=0 sane=1
+	idtentry_body noist_\cfunc, has_error_code=0
 
 _ASM_NOKPROBE(\asmsym)
 SYM_CODE_END(\asmsym)
arch/x86/include/asm/idtentry.h +3 -3
@@ -283,10 +283,10 @@ static __always_inline void __##func(str
  * The ASM variants for DECLARE_IDTENTRY*() which emit the ASM entry stubs.
  */
 #define DECLARE_IDTENTRY(vector, func)					\
-	idtentry vector asm_##func func has_error_code=0 sane=1
+	idtentry vector asm_##func func has_error_code=0
 
 #define DECLARE_IDTENTRY_ERRORCODE(vector, func)			\
-	idtentry vector asm_##func func has_error_code=1 sane=1
+	idtentry vector asm_##func func has_error_code=1
 
 /* Special case for 32bit IRET 'trap'. Do not emit ASM code */
 #define DECLARE_IDTENTRY_SW(vector, func)
@@ -324,7 +324,7 @@ static __always_inline void __##func(str
 
 /* XEN NMI and DB wrapper */
 #define DECLARE_IDTENTRY_XEN(vector, func)				\
-	idtentry vector asm_exc_xen##func exc_##func has_error_code=0 sane=1
+	idtentry vector asm_exc_xen##func exc_##func has_error_code=0
 
 #endif /* __ASSEMBLY__ */
 
#56 lore raw

[tip: x86/entry] x86/entry: Remove the transition leftovers

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     ae8f8f2116d86618cf43be142c495906a17811ef
Gitweb:        https://git.kernel.org/tip/ae8f8f2116d86618cf43be142c495906a17811ef
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:29 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Remove the transition leftovers

Now that all exceptions are converted over the sane flag is not longer needed. Also the vector argument of idtentry_body on 64-bit is pointless now.

---
 arch/x86/entry/entry_32.S       |  3 +--
 arch/x86/entry/entry_64.S       | 26 ++++----------------------
 arch/x86/include/asm/idtentry.h |  6 +++---
 3 files changed, 8 insertions(+), 27 deletions(-)
arch/x86/entry/entry_32.S +1 -2
index 1674deb..8b29330 100644
@@ -734,9 +734,8 @@
  * @asmsym:		ASM symbol for the entry point
  * @cfunc:		C function to be called
  * @has_error_code:	Hardware pushed error code on stack
- * @sane:		Compatibility flag with 64bit
  */
-.macro idtentry vector asmsym cfunc has_error_code:req sane=0
+.macro idtentry vector asmsym cfunc has_error_code:req
 SYM_CODE_START(\asmsym)
 	ASM_CLAC
 	cld
arch/x86/entry/entry_64.S +4 -22
index 5789f76..2e476f4 100644
@@ -496,27 +496,14 @@ SYM_CODE_END(spurious_entries_start)
 
 /**
  * idtentry_body - Macro to emit code calling the C function
- * @vector:		Vector number
  * @cfunc:		C function to be called
  * @has_error_code:	Hardware pushed error code on stack
- * @sane:		Sane variant which handles irq tracing, context tracking in C
  */
-.macro idtentry_body vector cfunc has_error_code:req sane=0
+.macro idtentry_body cfunc has_error_code:req
 
 	call	error_entry
 	UNWIND_HINT_REGS
 
-	.if \sane == 0
-	TRACE_IRQS_OFF
-
-#ifdef CONFIG_CONTEXT_TRACKING
-	testb	$3, CS(%rsp)
-	jz	.Lfrom_kernel_no_ctxt_tracking_\@
-	CALL_enter_from_user_mode
-.Lfrom_kernel_no_ctxt_tracking_\@:
-#endif
-	.endif
-
 	movq	%rsp, %rdi			/* pt_regs pointer into 1st argument*/
 
 	.if \has_error_code == 1
@@ -526,11 +513,7 @@ SYM_CODE_END(spurious_entries_start)
 
 	call	\cfunc
 
-	.if \sane == 0
-	jmp	error_exit
-	.else
 	jmp	error_return
-	.endif
 .endm
 
 /**
@@ -539,12 +522,11 @@ SYM_CODE_END(spurious_entries_start)
  * @asmsym:		ASM symbol for the entry point
  * @cfunc:		C function to be called
  * @has_error_code:	Hardware pushed error code on stack
- * @sane:		Sane variant which handles irq tracing, context tracking in C
  *
  * The macro emits code to set up the kernel context for straight forward
  * and simple IDT entries. No IST stack, no paranoid entry checks.
  */
-.macro idtentry vector asmsym cfunc has_error_code:req sane=0
+.macro idtentry vector asmsym cfunc has_error_code:req
 SYM_CODE_START(\asmsym)
 	UNWIND_HINT_IRET_REGS offset=\has_error_code*8
 	ASM_CLAC
@@ -567,7 +549,7 @@ SYM_CODE_START(\asmsym)
 .Lfrom_usermode_no_gap_\@:
 	.endif
 
-	idtentry_body \vector \cfunc \has_error_code \sane
+	idtentry_body \cfunc \has_error_code
 
 _ASM_NOKPROBE(\asmsym)
 SYM_CODE_END(\asmsym)
@@ -642,7 +624,7 @@ SYM_CODE_START(\asmsym)
 
 	/* Switch to the regular task stack and use the noist entry point */
 .Lfrom_usermode_switch_stack_\@:
-	idtentry_body vector noist_\cfunc, has_error_code=0 sane=1
+	idtentry_body noist_\cfunc, has_error_code=0
 
 _ASM_NOKPROBE(\asmsym)
 SYM_CODE_END(\asmsym)
arch/x86/include/asm/idtentry.h +3 -3
index 92054ff..53e7f14 100644
@@ -283,10 +283,10 @@ __visible noinstr void func(struct pt_regs *regs,			\
  * The ASM variants for DECLARE_IDTENTRY*() which emit the ASM entry stubs.
  */
 #define DECLARE_IDTENTRY(vector, func)					\
-	idtentry vector asm_##func func has_error_code=0 sane=1
+	idtentry vector asm_##func func has_error_code=0
 
 #define DECLARE_IDTENTRY_ERRORCODE(vector, func)			\
-	idtentry vector asm_##func func has_error_code=1 sane=1
+	idtentry vector asm_##func func has_error_code=1
 
 /* Special case for 32bit IRET 'trap'. Do not emit ASM code */
 #define DECLARE_IDTENTRY_SW(vector, func)
@@ -324,7 +324,7 @@ __visible noinstr void func(struct pt_regs *regs,			\
 
 /* XEN NMI and DB wrapper */
 #define DECLARE_IDTENTRY_XEN(vector, func)				\
-	idtentry vector asm_exc_xen##func exc_##func has_error_code=0 sane=1
+	idtentry vector asm_exc_xen##func exc_##func has_error_code=0
 
 #endif /* __ASSEMBLY__ */
 
#57 lore raw

PATCH v9 17/39 x86/entry: Change exit path of xen_failsafe_callback

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

xen_failsafe_callback is invoked from XEN for two cases:

1. Fault while reloading DS, ES, FS or GS
2. Fault while executing IRET

#1 retries the IRET after XEN has fixed up the segments. #2 injects a #GP which kills the task

For #1 there is no reason to go through the full exception return path because the tasks TIF state is still the same. So just going straight to the IRET path is good enough.

---
 arch/x86/entry/entry_32.S |    2 +-
 arch/x86/entry/entry_64.S |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/entry/entry_32.S +1 -1
@@ -1352,7 +1352,7 @@ SYM_FUNC_START(xen_failsafe_callback)
 5:	pushl	$-1				/* orig_ax = -1 => not a system call */
 	SAVE_ALL
 	ENCODE_FRAME_POINTER
-	jmp	ret_from_exception
+	jmp	handle_exception_return
 
 .section .fixup, "ax"
 6:	xorl	%eax, %eax
arch/x86/entry/entry_64.S +1 -1
@@ -1175,7 +1175,7 @@ SYM_CODE_START(xen_failsafe_callback)
 	pushq	$-1 /* orig_ax = -1 => not a system call */
 	PUSH_AND_CLEAR_REGS
 	ENCODE_FRAME_POINTER
-	jmp	error_exit
+	jmp	error_return
 SYM_CODE_END(xen_failsafe_callback)
 #endif /* CONFIG_XEN_PV */
 
#58 lore raw

[tip: x86/entry] x86/entry: Change exit path of xen_failsafe_callback

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     c8af31cbb0650b2be9b6a2a9cdb3a2a0770d7417
Gitweb:        https://git.kernel.org/tip/c8af31cbb0650b2be9b6a2a9cdb3a2a0770d7417
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:30 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Change exit path of xen_failsafe_callback

xen_failsafe_callback() is invoked from XEN for two cases:

1. Fault while reloading DS, ES, FS or GS
2. Fault while executing IRET

#1 retries the IRET after XEN has fixed up the segments. #2 injects a #GP which kills the task

For #1 there is no reason to go through the full exception return path because the tasks TIF state is still the same. So just going straight to the IRET path is good enough.

---
 arch/x86/entry/entry_32.S | 2 +-
 arch/x86/entry/entry_64.S | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/entry/entry_32.S +1 -1
index 8b29330..bd25706 100644
@@ -1352,7 +1352,7 @@ SYM_FUNC_START(xen_failsafe_callback)
 5:	pushl	$-1				/* orig_ax = -1 => not a system call */
 	SAVE_ALL
 	ENCODE_FRAME_POINTER
-	jmp	ret_from_exception
+	jmp	handle_exception_return
 
 .section .fixup, "ax"
 6:	xorl	%eax, %eax
arch/x86/entry/entry_64.S +1 -1
index 2e476f4..a526fb5 100644
@@ -1175,7 +1175,7 @@ SYM_CODE_START(xen_failsafe_callback)
 	pushq	$-1 /* orig_ax = -1 => not a system call */
 	PUSH_AND_CLEAR_REGS
 	ENCODE_FRAME_POINTER
-	jmp	error_exit
+	jmp	error_return
 SYM_CODE_END(xen_failsafe_callback)
 #endif /* CONFIG_XEN_PV */
 
#59 lore raw

PATCH v9 18/39 x86/entry/64: Remove error_exit

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

No more users.

---
 arch/x86/entry/entry_64.S |    9 ---------
 1 file changed, 9 deletions(-)
arch/x86/entry/entry_64.S +0 -9
@@ -1356,15 +1356,6 @@ SYM_CODE_START_LOCAL(error_entry)
 	jmp	.Lerror_entry_from_usermode_after_swapgs
 SYM_CODE_END(error_entry)
 
-SYM_CODE_START_LOCAL(error_exit)
-	UNWIND_HINT_REGS
-	DISABLE_INTERRUPTS(CLBR_ANY)
-	TRACE_IRQS_OFF
-	testb	$3, CS(%rsp)
-	jz	retint_kernel
-	jmp	.Lretint_user
-SYM_CODE_END(error_exit)
-
 SYM_CODE_START_LOCAL(error_return)
 	UNWIND_HINT_REGS
 	DEBUG_ENTRY_ASSERT_IRQS_OFF
#60 lore raw

[tip: x86/entry] x86/entry/64: Remove error_exit()

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     88d41f530736bdbed258df357e97956e8f97856d
Gitweb:        https://git.kernel.org/tip/88d41f530736bdbed258df357e97956e8f97856d
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:31 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry/64: Remove error_exit()

No more users.

---
 arch/x86/entry/entry_64.S |  9 ---------
 1 file changed, 9 deletions(-)
arch/x86/entry/entry_64.S +0 -9
index a526fb5..7699359 100644
@@ -1356,15 +1356,6 @@ SYM_CODE_START_LOCAL(error_entry)
 	jmp	.Lerror_entry_from_usermode_after_swapgs
 SYM_CODE_END(error_entry)
 
-SYM_CODE_START_LOCAL(error_exit)
-	UNWIND_HINT_REGS
-	DISABLE_INTERRUPTS(CLBR_ANY)
-	TRACE_IRQS_OFF
-	testb	$3, CS(%rsp)
-	jz	retint_kernel
-	jmp	.Lretint_user
-SYM_CODE_END(error_exit)
-
 SYM_CODE_START_LOCAL(error_return)
 	UNWIND_HINT_REGS
 	DEBUG_ENTRY_ASSERT_IRQS_OFF
#61 lore raw

PATCH v9 19/39 x86/entry/32: Remove common_exception

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

No more users.

---
 arch/x86/entry/entry_32.S |   21 ---------------------
 1 file changed, 21 deletions(-)
arch/x86/entry/entry_32.S +0 -21
@@ -1394,27 +1394,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vec
 
 #endif /* CONFIG_HYPERV */
 
-SYM_CODE_START_LOCAL_NOALIGN(common_exception)
-	/* the function address is in %gs's slot on the stack */
-	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
-	ENCODE_FRAME_POINTER
-
-	/* fixup %gs */
-	GS_TO_REG %ecx
-	movl	PT_GS(%esp), %edi		# get the function address
-	REG_TO_PTGS %ecx
-	SET_KERNEL_GS %ecx
-
-	/* fixup orig %eax */
-	movl	PT_ORIG_EAX(%esp), %edx		# get the error code
-	movl	$-1, PT_ORIG_EAX(%esp)		# no syscall to restart
-
-	TRACE_IRQS_OFF
-	movl	%esp, %eax			# pt_regs pointer
-	CALL_NOSPEC edi
-	jmp	ret_from_exception
-SYM_CODE_END(common_exception)
-
 SYM_CODE_START_LOCAL_NOALIGN(handle_exception)
 	/* the function address is in %gs's slot on the stack */
 	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
#62 lore raw

[tip: x86/entry] x86/entry/32: Remove common_exception()

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     b890361ba0f4a8eb565f2d8c6e045a70abc365eb
Gitweb:        https://git.kernel.org/tip/b890361ba0f4a8eb565f2d8c6e045a70abc365eb
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:32 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry/32: Remove common_exception()

No more users.

---
 arch/x86/entry/entry_32.S | 21 ---------------------
 1 file changed, 21 deletions(-)
arch/x86/entry/entry_32.S +0 -21
index bd25706..1bf2dcb 100644
@@ -1394,27 +1394,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR,
 
 #endif /* CONFIG_HYPERV */
 
-SYM_CODE_START_LOCAL_NOALIGN(common_exception)
-	/* the function address is in %gs's slot on the stack */
-	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
-	ENCODE_FRAME_POINTER
-
-	/* fixup %gs */
-	GS_TO_REG %ecx
-	movl	PT_GS(%esp), %edi		# get the function address
-	REG_TO_PTGS %ecx
-	SET_KERNEL_GS %ecx
-
-	/* fixup orig %eax */
-	movl	PT_ORIG_EAX(%esp), %edx		# get the error code
-	movl	$-1, PT_ORIG_EAX(%esp)		# no syscall to restart
-
-	TRACE_IRQS_OFF
-	movl	%esp, %eax			# pt_regs pointer
-	CALL_NOSPEC edi
-	jmp	ret_from_exception
-SYM_CODE_END(common_exception)
-
 SYM_CODE_START_LOCAL_NOALIGN(handle_exception)
 	/* the function address is in %gs's slot on the stack */
 	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
#63 lore raw

PATCH v9 20/39 x86/irq: Use generic irq_regs implementation

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

The only difference is the name of the per-CPU variable: irq_regs vs. __irq_regs, but the accessor functions are identical.

Remove the pointless copy and use the generic variant.

---
 arch/x86/include/asm/irq_regs.h |   32 --------------------------------
 arch/x86/kernel/irq.c           |    3 ---
 2 files changed, 35 deletions(-)
arch/x86/include/asm/irq_regs.h deleted +0 -32
@@ -1,32 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Per-cpu current frame pointer - the location of the last exception frame on
- * the stack, stored in the per-cpu area.
- *
- * Jeremy Fitzhardinge <[email protected]>
- */
-#ifndef _ASM_X86_IRQ_REGS_H
-#define _ASM_X86_IRQ_REGS_H
-
-#include <asm/percpu.h>
-
-#define ARCH_HAS_OWN_IRQ_REGS
-
-DECLARE_PER_CPU(struct pt_regs *, irq_regs);
-
-static inline struct pt_regs *get_irq_regs(void)
-{
-	return __this_cpu_read(irq_regs);
-}
-
-static inline struct pt_regs *set_irq_regs(struct pt_regs *new_regs)
-{
-	struct pt_regs *old_regs;
-
-	old_regs = get_irq_regs();
-	__this_cpu_write(irq_regs, new_regs);
-
-	return old_regs;
-}
-
-#endif /* _ASM_X86_IRQ_REGS_32_H */
arch/x86/kernel/irq.c +0 -3
@@ -26,9 +26,6 @@
 DEFINE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat);
 EXPORT_PER_CPU_SYMBOL(irq_stat);
 
-DEFINE_PER_CPU(struct pt_regs *, irq_regs);
-EXPORT_PER_CPU_SYMBOL(irq_regs);
-
 atomic_t irq_err_count;
 
 /*
#64 lore raw

Re: [patch V9 20/39] x86/irq: Use generic irq_regs implementation

Hello Thomas,

with this patch i got the following build error:

LD .tmp_vmlinux.kallsyms1
/home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function `get_irq_regs':
/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:19: undefined reference to irq_regs' /home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function set_irq_regs':
/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:27: undefined reference to `irq_regs' /home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: /home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:27: undefined reference to irq_regs' /home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function get_irq_regs':
/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:19: undefined reference to irq_regs' /home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function set_irq_regs':
/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:27: undefined reference to `irq_regs'

Best regards
Damian

#65 lore raw

Re: [patch V9 20/39] x86/irq: Use generic irq_regs implementation

Damian,

Quoted 5 lines · damian

with this patch i got the following build error:

LD .tmp_vmlinux.kallsyms1
/home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function `get_irq_regs':
/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:19: undefined reference to `irq_regs'

Can you provide the .config please?

Thanks,

tglx

#66 lore raw

Re: [patch V9 20/39] x86/irq: Use generic irq_regs implementation

Quoted 9 lines · Thomas Gleixner

Damian,

Quoted 5 lines · damian

with this patch i got the following build error:

LD .tmp_vmlinux.kallsyms1
/home/damian/kernel/buildroot/output/host/bin/x86_64-buildroot-linux-uclibc-ld: arch/x86/kernel/irq.o: in function `get_irq_regs':
/home/damian/kernel/buildroot/output/build/linux-custom/./arch/x86/include/asm/irq_regs.h:19: undefined reference to `irq_regs'

Quoted 5 lines

Can you provide the .config please?

Thanks,

tglx

Hello Thomas,

I found out self the reason. I rebuild the kernel from scratch again and now it works.

Many Thnaks
Damian

#67 lore raw

[tip: x86/entry] x86/irq: Use generic irq_regs implementation

In reply to Thomas Gleixner
The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     359763a015a4bc3ac9899bf8a3ea97dae91c8e2a
Gitweb:        https://git.kernel.org/tip/359763a015a4bc3ac9899bf8a3ea97dae91c8e2a
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:33 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/irq: Use generic irq_regs implementation

The only difference is the name of the per-CPU variable: irq_regs vs. __irq_regs, but the accessor functions are identical.

Remove the pointless copy and use the generic variant.

---
 arch/x86/include/asm/irq_regs.h | 32 +--------------------------------
 arch/x86/kernel/irq.c           |  3 +---
 2 files changed, 35 deletions(-)
 delete mode 100644 arch/x86/include/asm/irq_regs.h
arch/x86/include/asm/irq_regs.h deleted +0 -32
deleted file mode 100644
index 187ce59..0000000
@@ -1,32 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * Per-cpu current frame pointer - the location of the last exception frame on
- * the stack, stored in the per-cpu area.
- *
- * Jeremy Fitzhardinge <[email protected]>
- */
-#ifndef _ASM_X86_IRQ_REGS_H
-#define _ASM_X86_IRQ_REGS_H
-
-#include <asm/percpu.h>
-
-#define ARCH_HAS_OWN_IRQ_REGS
-
-DECLARE_PER_CPU(struct pt_regs *, irq_regs);
-
-static inline struct pt_regs *get_irq_regs(void)
-{
-	return __this_cpu_read(irq_regs);
-}
-
-static inline struct pt_regs *set_irq_regs(struct pt_regs *new_regs)
-{
-	struct pt_regs *old_regs;
-
-	old_regs = get_irq_regs();
-	__this_cpu_write(irq_regs, new_regs);
-
-	return old_regs;
-}
-
-#endif /* _ASM_X86_IRQ_REGS_32_H */
arch/x86/kernel/irq.c +0 -3
index c7965ff..252065d 100644
@@ -26,9 +26,6 @@
 DEFINE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat);
 EXPORT_PER_CPU_SYMBOL(irq_stat);
 
-DEFINE_PER_CPU(struct pt_regs *, irq_regs);
-EXPORT_PER_CPU_SYMBOL(irq_regs);
-
 atomic_t irq_err_count;
 
 /*
#68 lore raw

PATCH v9 21/39 x86/irq: Convey vector as argument and not in ptregs

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Device interrupts which go through do_IRQ() or the spurious interrupt handler have their separate entry code on 64 bit for no good reason.

Both 32 and 64 bit transport the vector number through ORIG_[RE]AX in pt_regs. Further the vector number is forced to fit into an u8 and is complemented and offset by 0x80 so it's in the signed character range. Otherwise GAS would expand the pushq to a 5 byte instruction for any vector > 0x7F.

Treat the vector number like an error code and hand it to the C function as argument. This allows to get rid of the extra entry code in a later step.

Simplify the error code push magic by implementing the pushq imm8 via a '.byte 0x6a, vector' sequence so GAS is not able to screw it up. As the pushq imm8 is sign extending the resulting error code needs to be truncated to 8 bits in C code.

---
V9: Add protection against GAS emitting more than 8 byte per vector

---
 arch/x86/entry/calling.h          |    5 +++-
 arch/x86/entry/entry_32.S         |   33 +++-------------------------
 arch/x86/entry/entry_64.S         |   40 ++++++----------------------------
 arch/x86/include/asm/entry_arch.h |    2 -
 arch/x86/include/asm/hw_irq.h     |    1 
 arch/x86/include/asm/idtentry.h   |   44 ++++++++++++++++++++++++++++++++++++++
 arch/x86/include/asm/irq.h        |    2 -
 arch/x86/include/asm/traps.h      |    3 +-
 arch/x86/kernel/apic/apic.c       |   31 +++++++++++++++++++++-----
 arch/x86/kernel/idt.c             |    2 -
 arch/x86/kernel/irq.c             |   14 +++++++-----
 11 files changed, 99 insertions(+), 78 deletions(-)
arch/x86/entry/calling.h +4 -1
@@ -341,7 +341,10 @@ For 32-bit we have the following convent
 #endif
 .endm
 
-#endif /* CONFIG_X86_64 */
+#else /* CONFIG_X86_64 */
+# undef		UNWIND_HINT_IRET_REGS
+# define	UNWIND_HINT_IRET_REGS
+#endif /* !CONFIG_X86_64 */
 
 .macro STACKLEAK_ERASE
 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK
arch/x86/entry/entry_32.S +4 -29
@@ -1215,40 +1215,15 @@ SYM_FUNC_END(entry_INT80_32)
 #endif
 .endm
 
-/*
- * Build the entry stubs with some assembler magic.
- * We pack 1 stub into every 8-byte block.
- */
-	.align 8
-SYM_CODE_START(irq_entries_start)
-    vector=FIRST_EXTERNAL_VECTOR
-    .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
-	pushl	$(~vector+0x80)			/* Note: always in signed byte range */
-    vector=vector+1
-	jmp	common_interrupt
-	.align	8
-    .endr
-SYM_CODE_END(irq_entries_start)
-
 #ifdef CONFIG_X86_LOCAL_APIC
-	.align 8
-SYM_CODE_START(spurious_entries_start)
-    vector=FIRST_SYSTEM_VECTOR
-    .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR)
-	pushl	$(~vector+0x80)			/* Note: always in signed byte range */
-    vector=vector+1
-	jmp	common_spurious
-	.align	8
-    .endr
-SYM_CODE_END(spurious_entries_start)
-
 SYM_CODE_START_LOCAL(common_spurious)
 	ASM_CLAC
-	addl	$-0x80, (%esp)			/* Adjust vector into the [-256, -1] range */
 	SAVE_ALL switch_stacks=1
 	ENCODE_FRAME_POINTER
 	TRACE_IRQS_OFF
 	movl	%esp, %eax
+	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
+	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
 	call	smp_spurious_interrupt
 	jmp	ret_from_intr
 SYM_CODE_END(common_spurious)
@@ -1261,12 +1236,12 @@ SYM_CODE_END(common_spurious)
 	.p2align CONFIG_X86_L1_CACHE_SHIFT
 SYM_CODE_START_LOCAL(common_interrupt)
 	ASM_CLAC
-	addl	$-0x80, (%esp)			/* Adjust vector into the [-256, -1] range */
-
 	SAVE_ALL switch_stacks=1
 	ENCODE_FRAME_POINTER
 	TRACE_IRQS_OFF
 	movl	%esp, %eax
+	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
+	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
 	call	do_IRQ
 	jmp	ret_from_intr
 SYM_CODE_END(common_interrupt)
arch/x86/entry/entry_64.S +7 -33
@@ -358,34 +358,6 @@ SYM_CODE_START(ret_from_fork)
 SYM_CODE_END(ret_from_fork)
 .popsection
 
-/*
- * Build the entry stubs with some assembler magic.
- * We pack 1 stub into every 8-byte block.
- */
-	.align 8
-SYM_CODE_START(irq_entries_start)
-    vector=FIRST_EXTERNAL_VECTOR
-    .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
-	UNWIND_HINT_IRET_REGS
-	pushq	$(~vector+0x80)			/* Note: always in signed byte range */
-	jmp	common_interrupt
-	.align	8
-	vector=vector+1
-    .endr
-SYM_CODE_END(irq_entries_start)
-
-	.align 8
-SYM_CODE_START(spurious_entries_start)
-    vector=FIRST_SYSTEM_VECTOR
-    .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR)
-	UNWIND_HINT_IRET_REGS
-	pushq	$(~vector+0x80)			/* Note: always in signed byte range */
-	jmp	common_spurious
-	.align	8
-	vector=vector+1
-    .endr
-SYM_CODE_END(spurious_entries_start)
-
 .macro DEBUG_ENTRY_ASSERT_IRQS_OFF
 #ifdef CONFIG_DEBUG_ENTRY
 	pushq %rax
@@ -755,13 +727,14 @@ SYM_CODE_END(interrupt_entry)
 /* Interrupt entry/exit. */
 
 /*
- * The interrupt stubs push (~vector+0x80) onto the stack and
+ * The interrupt stubs push vector onto the stack and
  * then jump to common_spurious/interrupt.
  */
 SYM_CODE_START_LOCAL(common_spurious)
-	addq	$-0x80, (%rsp)			/* Adjust vector to [-256, -1] range */
 	call	interrupt_entry
 	UNWIND_HINT_REGS indirect=1
+	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
+	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
 	call	smp_spurious_interrupt		/* rdi points to pt_regs */
 	jmp	ret_from_intr
 SYM_CODE_END(common_spurious)
@@ -770,10 +743,11 @@ SYM_CODE_END(common_spurious)
 /* common_interrupt is a hotpath. Align it */
 	.p2align CONFIG_X86_L1_CACHE_SHIFT
 SYM_CODE_START_LOCAL(common_interrupt)
-	addq	$-0x80, (%rsp)			/* Adjust vector to [-256, -1] range */
 	call	interrupt_entry
 	UNWIND_HINT_REGS indirect=1
-	call	do_IRQ	/* rdi points to pt_regs */
+	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
+	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
+	call	do_IRQ				/* rdi points to pt_regs */
 	/* 0(%rsp): old RSP */
 ret_from_intr:
 	DISABLE_INTERRUPTS(CLBR_ANY)
@@ -1022,7 +996,7 @@ apicinterrupt RESCHEDULE_VECTOR			resche
 #endif
 
 apicinterrupt ERROR_APIC_VECTOR			error_interrupt			smp_error_interrupt
-apicinterrupt SPURIOUS_APIC_VECTOR		spurious_interrupt		smp_spurious_interrupt
+apicinterrupt SPURIOUS_APIC_VECTOR		spurious_apic_interrupt		smp_spurious_apic_interrupt
 
 #ifdef CONFIG_IRQ_WORK
 apicinterrupt IRQ_WORK_VECTOR			irq_work_interrupt		smp_irq_work_interrupt
arch/x86/include/asm/entry_arch.h +1 -1
@@ -35,7 +35,7 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_i
 
 BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR)
 BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR)
-BUILD_INTERRUPT(spurious_interrupt,SPURIOUS_APIC_VECTOR)
+BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR)
 BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR)
 
 #ifdef CONFIG_IRQ_WORK
arch/x86/include/asm/hw_irq.h +1 -0
@@ -39,6 +39,7 @@ extern asmlinkage void irq_work_interrup
 extern asmlinkage void uv_bau_message_intr1(void);
 
 extern asmlinkage void spurious_interrupt(void);
+extern asmlinkage void spurious_apic_interrupt(void);
 extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
 
arch/x86/include/asm/idtentry.h +44 -0

50 lines not on this page · raw message

arch/x86/include/asm/irq.h +1 -1
@@ -36,7 +36,7 @@ extern void native_init_IRQ(void);
 
 extern void handle_irq(struct irq_desc *desc, struct pt_regs *regs);
 
-extern __visible void do_IRQ(struct pt_regs *regs);
+extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);
 
 extern void init_ISA_irqs(void);
 
arch/x86/include/asm/traps.h +2 -1
@@ -41,8 +41,9 @@ asmlinkage void smp_deferred_error_inter
 #endif
 
 void smp_apic_timer_interrupt(struct pt_regs *regs);
-void smp_spurious_interrupt(struct pt_regs *regs);
 void smp_error_interrupt(struct pt_regs *regs);
+void smp_spurious_apic_interrupt(struct pt_regs *regs);
+void smp_spurious_interrupt(struct pt_regs *regs, unsigned long vector);
 asmlinkage void smp_irq_move_cleanup_interrupt(void);
 
 #ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +25 -6
@@ -2153,15 +2153,29 @@ void __init register_lapic_address(unsig
  * Local APIC interrupts
  */
 
-/*
- * This interrupt should _never_ happen with our APIC/SMP architecture
+/**
+ * smp_spurious_interrupt - Catch all for interrupts raised on unused vectors
+ * @regs:	Pointer to pt_regs on stack
+ * @error_code:	The vector number is in the lower 8 bits
+ *
+ * This is invoked from ASM entry code to catch all interrupts which
+ * trigger on an entry which is routed to the common_spurious idtentry
+ * point.
+ *
+ * Also called from smp_spurious_apic_interrupt().
  */
-__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs)
+__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs,
+						  unsigned long vector)
 {
-	u8 vector = ~regs->orig_ax;
 	u32 v;
 
 	entering_irq();
+	/*
+	 * The push in the entry ASM code which stores the vector number on
+	 * the stack in the error code slot is sign expanding. Just use the
+	 * lower 8 bits.
+	 */
+	vector &= 0xFF;
 	trace_spurious_apic_entry(vector);
 
 	inc_irq_stat(irq_spurious_count);
@@ -2182,11 +2196,11 @@ void __init register_lapic_address(unsig
 	 */
 	v = apic_read(APIC_ISR + ((vector & ~0x1f) >> 1));
 	if (v & (1 << (vector & 0x1f))) {
-		pr_info("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n",
+		pr_info("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n",
 			vector, smp_processor_id());
 		ack_APIC_irq();
 	} else {
-		pr_info("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n",
+		pr_info("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n",
 			vector, smp_processor_id());
 	}
 out:
@@ -2194,6 +2208,11 @@ void __init register_lapic_address(unsig
 	exiting_irq();
 }
 
+__visible void smp_spurious_apic_interrupt(struct pt_regs *regs)
+{
+	smp_spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+}
+
 /*
  * This interrupt should never happen with our APIC/SMP architecture
  */
arch/x86/kernel/idt.c +1 -1
@@ -145,7 +145,7 @@ static const __initconst struct idt_data
 #ifdef CONFIG_X86_UV
 	INTG(UV_BAU_MESSAGE,		uv_bau_message_intr1),
 #endif
-	INTG(SPURIOUS_APIC_VECTOR,	spurious_interrupt),
+	INTG(SPURIOUS_APIC_VECTOR,	spurious_apic_interrupt),
 	INTG(ERROR_APIC_VECTOR,		error_interrupt),
 #endif
 };
arch/x86/kernel/irq.c +9 -5
@@ -227,14 +227,18 @@ u64 arch_irq_stat(void)
  * SMP cross-CPU interrupts have their own specific
  * handlers).
  */
-__visible void __irq_entry do_IRQ(struct pt_regs *regs)
+__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
-	struct irq_desc * desc;
-	/* high bit used in ret_from_ code  */
-	unsigned vector = ~regs->orig_ax;
+	struct irq_desc *desc;
 
 	entering_irq();
+	/*
+	 * The push in the entry ASM code which stores the vector number on
+	 * the stack in the error code slot is sign expanding. Just use the
+	 * lower 8 bits.
+	 */
+	vector &= 0xFF;
 
 	/* entering_irq() tells RCU that we're not quiescent.  Check it. */
 	RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
@@ -249,7 +253,7 @@ u64 arch_irq_stat(void)
 		ack_APIC_irq();
 
 		if (desc == VECTOR_UNUSED) {
-			pr_emerg_ratelimited("%s: %d.%d No irq handler for vector\n",
+			pr_emerg_ratelimited("%s: %d.%lu No irq handler for vector\n",
 					     __func__, smp_processor_id(),
 					     vector);
 		} else {
#69 lore raw

Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs

Review

Quoted 9 lines · Thomas Gleixner
(quoted diff) +9 -0
+	.align 8
+SYM_CODE_START(irq_entries_start)
+    vector=FIRST_EXTERNAL_VECTOR
+    pos = .
+    .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
+	UNWIND_HINT_IRET_REGS
+	.byte	0x6a, vector
+	jmp	common_interrupt
+	.nops (pos + 8) - .

Boris pointed out that .nops requires binutils >=2.31 ... Sigh!

The below delta patch works for the purpose.

Thanks,

tglx

arch/x86/include/asm/idtentry.h +6 -2
@@ -475,7 +475,9 @@ SYM_CODE_START(irq_entries_start)
 	UNWIND_HINT_IRET_REGS
 	.byte	0x6a, vector
 	jmp	asm_common_interrupt
-	.nops (pos + 8) - .
+	nop
+	/* Ensure that the above is 8 bytes max */
+	. = pos + 8
     pos=pos+8
     vector=vector+1
     .endr
@@ -490,7 +492,9 @@ SYM_CODE_START(spurious_entries_start)
 	UNWIND_HINT_IRET_REGS
 	.byte	0x6a, vector
 	jmp	asm_spurious_interrupt
-	.nops (pos + 8) - .
+	nop
+	/* Ensure that the above is 8 bytes max */
+	. = pos + 8
     pos=pos+8
     vector=vector+1
     .endr
#70 lore raw

[tip: x86/entry] x86/irq: Convey vector as argument and not in ptregs

In reply to Thomas Gleixner
The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     2f83c60d35e049112189fb9e30c346c1b97c70e4
Gitweb:        https://git.kernel.org/tip/2f83c60d35e049112189fb9e30c346c1b97c70e4
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:34 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/irq: Convey vector as argument and not in ptregs

Device interrupts which go through do_IRQ() or the spurious interrupt handler have their separate entry code on 64 bit for no good reason.

Both 32 and 64 bit transport the vector number through ORIG_[RE]AX in pt_regs. Further the vector number is forced to fit into an u8 and is complemented and offset by 0x80 so it's in the signed character range. Otherwise GAS would expand the pushq to a 5 byte instruction for any vector > 0x7F.

Treat the vector number like an error code and hand it to the C function as argument. This allows to get rid of the extra entry code in a later step.

Simplify the error code push magic by implementing the pushq imm8 via a '.byte 0x6a, vector' sequence so GAS is not able to screw it up. As the pushq imm8 is sign extending the resulting error code needs to be truncated to 8 bits in C code.

---
 arch/x86/entry/calling.h          |  5 ++-
 arch/x86/entry/entry_32.S         | 33 ++-------------------
 arch/x86/entry/entry_64.S         | 40 ++++---------------------
 arch/x86/include/asm/entry_arch.h |  2 +-
 arch/x86/include/asm/hw_irq.h     |  1 +-
 arch/x86/include/asm/idtentry.h   | 48 ++++++++++++++++++++++++++++++-
 arch/x86/include/asm/irq.h        |  2 +-
 arch/x86/include/asm/traps.h      |  3 +-
 arch/x86/kernel/apic/apic.c       | 31 +++++++++++++++----
 arch/x86/kernel/idt.c             |  2 +-
 arch/x86/kernel/irq.c             | 14 +++++----
 11 files changed, 103 insertions(+), 78 deletions(-)
arch/x86/entry/calling.h +4 -1
index 1c7f13b..98da0d3 100644
@@ -341,7 +341,10 @@ For 32-bit we have the following conventions - kernel is built with
 #endif
 .endm
 
-#endif /* CONFIG_X86_64 */
+#else /* CONFIG_X86_64 */
+# undef		UNWIND_HINT_IRET_REGS
+# define	UNWIND_HINT_IRET_REGS
+#endif /* !CONFIG_X86_64 */
 
 .macro STACKLEAK_ERASE
 #ifdef CONFIG_GCC_PLUGIN_STACKLEAK
arch/x86/entry/entry_32.S +4 -29
index 1bf2dcb..53a6447 100644
@@ -1215,40 +1215,15 @@ SYM_FUNC_END(entry_INT80_32)
 #endif
 .endm
 
-/*
- * Build the entry stubs with some assembler magic.
- * We pack 1 stub into every 8-byte block.
- */
-	.align 8
-SYM_CODE_START(irq_entries_start)
-    vector=FIRST_EXTERNAL_VECTOR
-    .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
-	pushl	$(~vector+0x80)			/* Note: always in signed byte range */
-    vector=vector+1
-	jmp	common_interrupt
-	.align	8
-    .endr
-SYM_CODE_END(irq_entries_start)
-
 #ifdef CONFIG_X86_LOCAL_APIC
-	.align 8
-SYM_CODE_START(spurious_entries_start)
-    vector=FIRST_SYSTEM_VECTOR
-    .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR)
-	pushl	$(~vector+0x80)			/* Note: always in signed byte range */
-    vector=vector+1
-	jmp	common_spurious
-	.align	8
-    .endr
-SYM_CODE_END(spurious_entries_start)
-
 SYM_CODE_START_LOCAL(common_spurious)
 	ASM_CLAC
-	addl	$-0x80, (%esp)			/* Adjust vector into the [-256, -1] range */
 	SAVE_ALL switch_stacks=1
 	ENCODE_FRAME_POINTER
 	TRACE_IRQS_OFF
 	movl	%esp, %eax
+	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
+	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
 	call	smp_spurious_interrupt
 	jmp	ret_from_intr
 SYM_CODE_END(common_spurious)
@@ -1261,12 +1236,12 @@ SYM_CODE_END(common_spurious)
 	.p2align CONFIG_X86_L1_CACHE_SHIFT
 SYM_CODE_START_LOCAL(common_interrupt)
 	ASM_CLAC
-	addl	$-0x80, (%esp)			/* Adjust vector into the [-256, -1] range */
-
 	SAVE_ALL switch_stacks=1
 	ENCODE_FRAME_POINTER
 	TRACE_IRQS_OFF
 	movl	%esp, %eax
+	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
+	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
 	call	do_IRQ
 	jmp	ret_from_intr
 SYM_CODE_END(common_interrupt)
arch/x86/entry/entry_64.S +7 -33
index 7699359..e7434cd 100644
@@ -358,34 +358,6 @@ SYM_CODE_START(ret_from_fork)
 SYM_CODE_END(ret_from_fork)
 .popsection
 
-/*
- * Build the entry stubs with some assembler magic.
- * We pack 1 stub into every 8-byte block.
- */
-	.align 8
-SYM_CODE_START(irq_entries_start)
-    vector=FIRST_EXTERNAL_VECTOR
-    .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
-	UNWIND_HINT_IRET_REGS
-	pushq	$(~vector+0x80)			/* Note: always in signed byte range */
-	jmp	common_interrupt
-	.align	8
-	vector=vector+1
-    .endr
-SYM_CODE_END(irq_entries_start)
-
-	.align 8
-SYM_CODE_START(spurious_entries_start)
-    vector=FIRST_SYSTEM_VECTOR
-    .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR)
-	UNWIND_HINT_IRET_REGS
-	pushq	$(~vector+0x80)			/* Note: always in signed byte range */
-	jmp	common_spurious
-	.align	8
-	vector=vector+1
-    .endr
-SYM_CODE_END(spurious_entries_start)
-
 .macro DEBUG_ENTRY_ASSERT_IRQS_OFF
 #ifdef CONFIG_DEBUG_ENTRY
 	pushq %rax
@@ -755,13 +727,14 @@ _ASM_NOKPROBE(interrupt_entry)
 /* Interrupt entry/exit. */
 
 /*
- * The interrupt stubs push (~vector+0x80) onto the stack and
+ * The interrupt stubs push vector onto the stack and
  * then jump to common_spurious/interrupt.
  */
 SYM_CODE_START_LOCAL(common_spurious)
-	addq	$-0x80, (%rsp)			/* Adjust vector to [-256, -1] range */
 	call	interrupt_entry
 	UNWIND_HINT_REGS indirect=1
+	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
+	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
 	call	smp_spurious_interrupt		/* rdi points to pt_regs */
 	jmp	ret_from_intr
 SYM_CODE_END(common_spurious)
@@ -770,10 +743,11 @@ _ASM_NOKPROBE(common_spurious)
 /* common_interrupt is a hotpath. Align it */
 	.p2align CONFIG_X86_L1_CACHE_SHIFT
 SYM_CODE_START_LOCAL(common_interrupt)
-	addq	$-0x80, (%rsp)			/* Adjust vector to [-256, -1] range */
 	call	interrupt_entry
 	UNWIND_HINT_REGS indirect=1
-	call	do_IRQ	/* rdi points to pt_regs */
+	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
+	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
+	call	do_IRQ				/* rdi points to pt_regs */
 	/* 0(%rsp): old RSP */
 ret_from_intr:
 	DISABLE_INTERRUPTS(CLBR_ANY)
@@ -1022,7 +996,7 @@ apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
 
 apicinterrupt ERROR_APIC_VECTOR			error_interrupt			smp_error_interrupt
-apicinterrupt SPURIOUS_APIC_VECTOR		spurious_interrupt		smp_spurious_interrupt
+apicinterrupt SPURIOUS_APIC_VECTOR		spurious_apic_interrupt		smp_spurious_apic_interrupt
 
 #ifdef CONFIG_IRQ_WORK
 apicinterrupt IRQ_WORK_VECTOR			irq_work_interrupt		smp_irq_work_interrupt
arch/x86/include/asm/entry_arch.h +1 -1
index 4164227..cd57ce6 100644
@@ -35,7 +35,7 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)
 
 BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR)
 BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR)
-BUILD_INTERRUPT(spurious_interrupt,SPURIOUS_APIC_VECTOR)
+BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR)
 BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR)
 
 #ifdef CONFIG_IRQ_WORK
arch/x86/include/asm/hw_irq.h +1 -0
index 4154bc5..0ffe807 100644
@@ -39,6 +39,7 @@ extern asmlinkage void irq_work_interrupt(void);
 extern asmlinkage void uv_bau_message_intr1(void);
 
 extern asmlinkage void spurious_interrupt(void);
+extern asmlinkage void spurious_apic_interrupt(void);
 extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
 
arch/x86/include/asm/idtentry.h +48 -0

54 lines not on this page · raw message

arch/x86/include/asm/irq.h +1 -1
index 72fba0e..74690a3 100644
@@ -36,7 +36,7 @@ extern void native_init_IRQ(void);
 
 extern void handle_irq(struct irq_desc *desc, struct pt_regs *regs);
 
-extern __visible void do_IRQ(struct pt_regs *regs);
+extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);
 
 extern void init_ISA_irqs(void);
 
arch/x86/include/asm/traps.h +2 -1
index d7de360..32b2bec 100644
@@ -41,8 +41,9 @@ asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);
 #endif
 
 void smp_apic_timer_interrupt(struct pt_regs *regs);
-void smp_spurious_interrupt(struct pt_regs *regs);
 void smp_error_interrupt(struct pt_regs *regs);
+void smp_spurious_apic_interrupt(struct pt_regs *regs);
+void smp_spurious_interrupt(struct pt_regs *regs, unsigned long vector);
 asmlinkage void smp_irq_move_cleanup_interrupt(void);
 
 #ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +25 -6
index e53dda2..a4218a3 100644
@@ -2153,15 +2153,29 @@ void __init register_lapic_address(unsigned long address)
  * Local APIC interrupts
  */
 
-/*
- * This interrupt should _never_ happen with our APIC/SMP architecture
+/**
+ * smp_spurious_interrupt - Catch all for interrupts raised on unused vectors
+ * @regs:	Pointer to pt_regs on stack
+ * @error_code:	The vector number is in the lower 8 bits
+ *
+ * This is invoked from ASM entry code to catch all interrupts which
+ * trigger on an entry which is routed to the common_spurious idtentry
+ * point.
+ *
+ * Also called from smp_spurious_apic_interrupt().
  */
-__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs)
+__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs,
+						  unsigned long vector)
 {
-	u8 vector = ~regs->orig_ax;
 	u32 v;
 
 	entering_irq();
+	/*
+	 * The push in the entry ASM code which stores the vector number on
+	 * the stack in the error code slot is sign expanding. Just use the
+	 * lower 8 bits.
+	 */
+	vector &= 0xFF;
 	trace_spurious_apic_entry(vector);
 
 	inc_irq_stat(irq_spurious_count);
@@ -2182,11 +2196,11 @@ __visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs)
 	 */
 	v = apic_read(APIC_ISR + ((vector & ~0x1f) >> 1));
 	if (v & (1 << (vector & 0x1f))) {
-		pr_info("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n",
+		pr_info("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n",
 			vector, smp_processor_id());
 		ack_APIC_irq();
 	} else {
-		pr_info("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n",
+		pr_info("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n",
 			vector, smp_processor_id());
 	}
 out:
@@ -2194,6 +2208,11 @@ out:
 	exiting_irq();
 }
 
+__visible void smp_spurious_apic_interrupt(struct pt_regs *regs)
+{
+	smp_spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+}
+
 /*
  * This interrupt should never happen with our APIC/SMP architecture
  */
arch/x86/kernel/idt.c +1 -1
index ddb1115..20408e3 100644
@@ -145,7 +145,7 @@ static const __initconst struct idt_data apic_idts[] = {
 #ifdef CONFIG_X86_UV
 	INTG(UV_BAU_MESSAGE,		uv_bau_message_intr1),
 #endif
-	INTG(SPURIOUS_APIC_VECTOR,	spurious_interrupt),
+	INTG(SPURIOUS_APIC_VECTOR,	spurious_apic_interrupt),
 	INTG(ERROR_APIC_VECTOR,		error_interrupt),
 #endif
 };
arch/x86/kernel/irq.c +9 -5
index 252065d..c766936 100644
@@ -227,14 +227,18 @@ u64 arch_irq_stat(void)
  * SMP cross-CPU interrupts have their own specific
  * handlers).
  */
-__visible void __irq_entry do_IRQ(struct pt_regs *regs)
+__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
-	struct irq_desc * desc;
-	/* high bit used in ret_from_ code  */
-	unsigned vector = ~regs->orig_ax;
+	struct irq_desc *desc;
 
 	entering_irq();
+	/*
+	 * The push in the entry ASM code which stores the vector number on
+	 * the stack in the error code slot is sign expanding. Just use the
+	 * lower 8 bits.
+	 */
+	vector &= 0xFF;
 
 	/* entering_irq() tells RCU that we're not quiescent.  Check it. */
 	RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
@@ -249,7 +253,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs)
 		ack_APIC_irq();
 
 		if (desc == VECTOR_UNUSED) {
-			pr_emerg_ratelimited("%s: %d.%d No irq handler for vector\n",
+			pr_emerg_ratelimited("%s: %d.%lu No irq handler for vector\n",
 					     __func__, smp_processor_id(),
 					     vector);
 		} else {
#71 lore raw

Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs

In reply to Thomas Gleixner

Hi Thomas,

Quoted 14 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Device interrupts which go through do_IRQ() or the spurious interrupt handler have their separate entry code on 64 bit for no good reason.

Both 32 and 64 bit transport the vector number through ORIG_[RE]AX in pt_regs. Further the vector number is forced to fit into an u8 and is complemented and offset by 0x80 so it's in the signed character range. Otherwise GAS would expand the pushq to a 5 byte instruction for any vector > 0x7F.

Treat the vector number like an error code and hand it to the C function as argument. This allows to get rid of the extra entry code in a later step.

Quoted 8 lines

Simplify the error code push magic by implementing the pushq imm8 via a '.byte 0x6a, vector' sequence so GAS is not able to screw it up. As the pushq imm8 is sign extending the resulting error code needs to be truncated to 8 bits in C code.

I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:

https://github.com/scylladb/scylla/issues/7036

What we're seeing is that syscalls such as membarrier() take forever (0-10 µs would be normal):

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
53.26 12.458881 185953 67 membarrier

 15.79    3.693651       17843       207        49 read
 11.17    2.613350       67008        39           io_pgetevents
 10.89    2.547772       11795       216           timerfd_settime
  6.91    1.616802       11073       146           rt_sigprocmask
  1.39    0.325955        3542        92           timer_settime
  0.36    0.083691         526       159           io_submit
  0.22    0.051399         535        96           write
  0.00    0.000783          37        21           sendmsg
  0.00    0.000057           3        18         9 ioctl

------ ----------- ----------- --------- --------- ----------------
100.00 23.392341 1061 58 total

That again seems to stem from a vastly slowed down smp_call_function_many_cond():

Samples: 218K of event 'cpu-clock', 4000 Hz

Overhead  Shared Object        Symbol
  94.51%  [kernel]             [k] smp_call_function_many_cond
   0.76%  [kernel]             [k] __do_softirq
   0.32%  [kernel]             [k] native_queued_spin_lock_slowpath

[...]

which is stuck in

       │     csd_lock_wait():
       │             smp_cond_load_acquire(&csd->flags, !(VAL &
  0.00 │       mov    0x8(%rcx),%edx
  0.00 │       and    $0x1,%edx
       │     ↓ je     2b9
       │     rep_nop():
  0.70 │2af:   pause
       │     csd_lock_wait():
 92.82 │       mov    0x8(%rcx),%edx
  6.48 │       and    $0x1,%edx
  0.00 │     ↑ jne    2af
  0.00 │2b9: ↑ jmp    282

Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.

Do you have any further pointers I could look at?

Thanks,

Alex

Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

#72 lore raw

Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs

Alex,

Quoted 8 lines · Alexander Graf

I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:

   https://github.com/scylladb/scylla/issues/7036

What we're seeing is that syscalls such as membarrier() take forever (0-10 µs would be normal):

...

Quoted 26 lines

That again seems to stem from a vastly slowed down smp_call_function_many_cond():

Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol

   94.51%  [kernel]             [k] smp_call_function_many_cond
    0.76%  [kernel]             [k] __do_softirq
    0.32%  [kernel]             [k] native_queued_spin_lock_slowpath

[...]

which is stuck in

        │     csd_lock_wait():
        │             smp_cond_load_acquire(&csd->flags, !(VAL &
   0.00 │       mov    0x8(%rcx),%edx
   0.00 │       and    $0x1,%edx
        │     ↓ je     2b9
        │     rep_nop():
   0.70 │2af:   pause
        │     csd_lock_wait():
  92.82 │       mov    0x8(%rcx),%edx
   6.48 │       and    $0x1,%edx
   0.00 │     ↑ jne    2af
   0.00 │2b9: ↑ jmp    282
Quoted 2 lines

Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.

I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through do_IRQ() at all.

Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.

Thanks,

tglx

#73 lore raw

Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs

Hi Thomas,

Quoted 48 lines · Thomas Gleixner

Alex,

Quoted 8 lines · Alexander Graf

I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:

    https://github.com/scylladb/scylla/issues/7036

What we're seeing is that syscalls such as membarrier() take forever (0-10 µs would be normal):

...

Quoted 26 lines

That again seems to stem from a vastly slowed down smp_call_function_many_cond():

Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol

    94.51%  [kernel]             [k] smp_call_function_many_cond
     0.76%  [kernel]             [k] __do_softirq
     0.32%  [kernel]             [k] native_queued_spin_lock_slowpath

[...]

which is stuck in

         │     csd_lock_wait():
         │             smp_cond_load_acquire(&csd->flags, !(VAL &
    0.00 │       mov    0x8(%rcx),%edx
    0.00 │       and    $0x1,%edx
         │     ↓ je     2b9
         │     rep_nop():
    0.70 │2af:   pause
         │     csd_lock_wait():
   92.82 │       mov    0x8(%rcx),%edx
    6.48 │       and    $0x1,%edx
    0.00 │     ↑ jne    2af
    0.00 │2b9: ↑ jmp    282
Quoted 2 lines

Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.

I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through do_IRQ() at all.

Quoted 3 lines

Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.

I am as puzzled as you are, but the bisect is very clear: 79b9c183021e works fast and 633260fa1 (as well as mainline) shows the weird behavior above.

It gets even better. This small (demonstrative only, mangled) patch on top of 633260fa1 also resolves the performance issue:

arch/x86/kernel/irq.c +1 -0
index c766936..7e91e9a 100644
@@ -239,6 +239,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector)
 	 * lower 8 bits.
 	 */
 	vector &= 0xFF;
+	regs->orig_ax = ~vector;
 
 	/* entering_irq() tells RCU that we're not quiescent.  Check it. */
 	RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");

To me that sounds like some irq exit code somewhere must still be looking at orig_ax to decide on something - and that something is wrong now that we removed the negation of the vector. It also seems to have an impact on remote function calls.

I'll have a deeper look tomorrow again if I can find any such place, but I wouldn't mind if anyone could point me into the right direction earlier :).

Alex

Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

maintainer
#74 lore raw

Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs

Quoted 5 lines · Alexander Graf

Hi Thomas,

Quoted 1 line · Thomas Gleixner

void irq_complete_move(struct irq_cfg *cfg)

{
        __irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}
Quoted 73 lines
Quoted 47 lines

Alex,

Quoted 8 lines · Alexander Graf

I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:

    https://github.com/scylladb/scylla/issues/7036

What we're seeing is that syscalls such as membarrier() take forever (0-10 µs would be normal):

...

Quoted 26 lines

That again seems to stem from a vastly slowed down smp_call_function_many_cond():

Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol

    94.51%  [kernel]             [k] smp_call_function_many_cond
     0.76%  [kernel]             [k] __do_softirq
     0.32%  [kernel]             [k] native_queued_spin_lock_slowpath

[...]

which is stuck in

         │     csd_lock_wait():
         │             smp_cond_load_acquire(&csd->flags, !(VAL &
    0.00 │       mov    0x8(%rcx),%edx
    0.00 │       and    $0x1,%edx
         │     ↓ je     2b9
         │     rep_nop():
    0.70 │2af:   pause
         │     csd_lock_wait():
   92.82 │       mov    0x8(%rcx),%edx
    6.48 │       and    $0x1,%edx
    0.00 │     ↑ jne    2af
    0.00 │2b9: ↑ jmp    282
Quoted 2 lines

Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.

I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through do_IRQ() at all.

Quoted 3 lines

Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.

I am as puzzled as you are, but the bisect is very clear: 79b9c183021e works fast and 633260fa1 (as well as mainline) shows the weird behavior above.

It gets even better. This small (demonstrative only, mangled) patch on top of 633260fa1 also resolves the performance issue:

arch/x86/kernel/irq.c +1 -0
index c766936..7e91e9a 100644
@@ -239,6 +239,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs
 *regs, unsigned long vector)
         * lower 8 bits.
         */
        vector &= 0xFF;
+       regs->orig_ax = ~vector;
 
        /* entering_irq() tells RCU that we're not quiescent.  Check it. */
        RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
Quoted 8 lines

To me that sounds like some irq exit code somewhere must still be looking at orig_ax to decide on something - and that something is wrong now that we removed the negation of the vector. It also seems to have an impact on remote function calls.

I'll have a deeper look tomorrow again if I can find any such place, but I wouldn't mind if anyone could point me into the right direction earlier :).

How about this:

void irq_complete_move(struct irq_cfg *cfg)

{
        __irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}

in arch/x86/kernel/apic/vector.c.

#75 lore raw

Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs

Hi Andy,

Quoted 95 lines · Andy Lutomirski
Quoted 5 lines · Alexander Graf

Hi Thomas,

Quoted 1 line · Thomas Gleixner

void irq_complete_move(struct irq_cfg *cfg)

{
         __irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}
Quoted 73 lines
Quoted 47 lines

Alex,

Quoted 8 lines · Alexander Graf

I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:

     https://github.com/scylladb/scylla/issues/7036

What we're seeing is that syscalls such as membarrier() take forever (0-10 µs would be normal):

...

Quoted 26 lines

That again seems to stem from a vastly slowed down smp_call_function_many_cond():

Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol

     94.51%  [kernel]             [k] smp_call_function_many_cond
      0.76%  [kernel]             [k] __do_softirq
      0.32%  [kernel]             [k] native_queued_spin_lock_slowpath

[...]

which is stuck in

          │     csd_lock_wait():
          │             smp_cond_load_acquire(&csd->flags, !(VAL &
     0.00 │       mov    0x8(%rcx),%edx
     0.00 │       and    $0x1,%edx
          │     ↓ je     2b9
          │     rep_nop():
     0.70 │2af:   pause
          │     csd_lock_wait():
    92.82 │       mov    0x8(%rcx),%edx
     6.48 │       and    $0x1,%edx
     0.00 │     ↑ jne    2af
     0.00 │2b9: ↑ jmp    282
Quoted 2 lines

Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.

I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through do_IRQ() at all.

Quoted 3 lines

Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.

I am as puzzled as you are, but the bisect is very clear: 79b9c183021e works fast and 633260fa1 (as well as mainline) shows the weird behavior above.

It gets even better. This small (demonstrative only, mangled) patch on top of 633260fa1 also resolves the performance issue:

arch/x86/kernel/irq.c +1 -0
index c766936..7e91e9a 100644
@@ -239,6 +239,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs
 *regs, unsigned long vector)
          * lower 8 bits.
          */
         vector &= 0xFF;
+       regs->orig_ax = ~vector;
 
         /* entering_irq() tells RCU that we're not quiescent.  Check it. */
         RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
Quoted 8 lines

To me that sounds like some irq exit code somewhere must still be looking at orig_ax to decide on something - and that something is wrong now that we removed the negation of the vector. It also seems to have an impact on remote function calls.

I'll have a deeper look tomorrow again if I can find any such place, but I wouldn't mind if anyone could point me into the right direction earlier :).

How about this:

Quoted 7 lines

void irq_complete_move(struct irq_cfg *cfg)

{
         __irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}

in arch/x86/kernel/apic/vector.c.

Thanks a lot, I stumbled over the same thing just after I sent the email as well and had been trying to see if I can quickly patch it up before I fall asleep :).

The code path above is used by the APIC vector move (irqbalance) logic, which explains why not everyone was seeing issues.

So with 633260fa1 applied, we never get out of moving state for our IRQ because orig_ax is always -1. That means we send an IPI to the cleanup vector on every single device interrupt, completely occupying the poor CPU that we moved the IRQ from.

I've confirmed that the patch below fixes the issue and will send a proper, more complete patch on top of mainline with fancy description and stable tag tomorrow.

Alex

arch/x86/entry/entry_64.S +0 -2
index e7434cd..a474e6e 100644
@@ -734,7 +734,6 @@ SYM_CODE_START_LOCAL(common_spurious)
 	call	interrupt_entry
 	UNWIND_HINT_REGS indirect=1
 	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
-	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
 	call	smp_spurious_interrupt		/* rdi points to pt_regs */
 	jmp	ret_from_intr
 SYM_CODE_END(common_spurious)
@@ -746,7 +745,6 @@ SYM_CODE_START_LOCAL(common_interrupt)
 	call	interrupt_entry
 	UNWIND_HINT_REGS indirect=1
 	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
-	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
 	call	do_IRQ				/* rdi points to pt_regs */
 	/* 0(%rsp): old RSP */
 ret_from_intr:
arch/x86/kernel/apic/vector.c +1 -1
index 67768e5443..5b6f74e 100644
@@ -934,7 +934,7 @@ static void __irq_complete_move(struct irq_cfg *cfg, unsigned vector)
 
 void irq_complete_move(struct irq_cfg *cfg)
 {
-	__irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
+	__irq_complete_move(cfg, get_irq_regs()->orig_ax);
 }
 
 /*

Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879

#76 lore raw

Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs

Quoted 153 lines · Alexander Graf

Hi Andy,

Quoted 95 lines · Andy Lutomirski
Quoted 5 lines · Alexander Graf

Hi Thomas,

Quoted 1 line · Thomas Gleixner

void irq_complete_move(struct irq_cfg *cfg)

{
         __irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}
Quoted 73 lines
Quoted 47 lines

Alex,

Quoted 8 lines · Alexander Graf

I'm currently trying to understand a performance regression with ScyllaDB on i3en.3xlarge (KVM based VM on Skylake) which we reliably bisected down to this commit:

     https://github.com/scylladb/scylla/issues/7036

What we're seeing is that syscalls such as membarrier() take forever (0-10 µs would be normal):

...

Quoted 26 lines

That again seems to stem from a vastly slowed down smp_call_function_many_cond():

Samples: 218K of event 'cpu-clock', 4000 Hz
Overhead Shared Object Symbol

     94.51%  [kernel]             [k] smp_call_function_many_cond
      0.76%  [kernel]             [k] __do_softirq
      0.32%  [kernel]             [k] native_queued_spin_lock_slowpath

[...]

which is stuck in

          │     csd_lock_wait():
          │             smp_cond_load_acquire(&csd->flags, !(VAL &
     0.00 │       mov    0x8(%rcx),%edx
     0.00 │       and    $0x1,%edx
          │     ↓ je     2b9
          │     rep_nop():
     0.70 │2af:   pause
          │     csd_lock_wait():
    92.82 │       mov    0x8(%rcx),%edx
     6.48 │       and    $0x1,%edx
     0.00 │     ↑ jne    2af
     0.00 │2b9: ↑ jmp    282
Quoted 2 lines

Given the patch at hand I was expecting lost IPIs, but I can't quite see anything getting lost.

I have no idea how that patch should be related to IPI and smp function calls. It's changing the way how regular device interrupts and their spurious counterpart are handled and not the way how IPIs are handled. They are handled by direct vectors and do not go through do_IRQ() at all.

Quoted 3 lines

Aside of that the commit just changes the way how the interrupt vector of a regular device interrupt is stored and conveyed. The extra read and write on the cache hot stack is hardly related to anything IPI.

I am as puzzled as you are, but the bisect is very clear: 79b9c183021e works fast and 633260fa1 (as well as mainline) shows the weird behavior above.

It gets even better. This small (demonstrative only, mangled) patch on top of 633260fa1 also resolves the performance issue:

arch/x86/kernel/irq.c +1 -0
index c766936..7e91e9a 100644
@@ -239,6 +239,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs
 *regs, unsigned long vector)
          * lower 8 bits.
          */
         vector &= 0xFF;
+       regs->orig_ax = ~vector;
 
         /* entering_irq() tells RCU that we're not quiescent.  Check it. */
         RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
Quoted 8 lines

To me that sounds like some irq exit code somewhere must still be looking at orig_ax to decide on something - and that something is wrong now that we removed the negation of the vector. It also seems to have an impact on remote function calls.

I'll have a deeper look tomorrow again if I can find any such place, but I wouldn't mind if anyone could point me into the right direction earlier :).

How about this:

Quoted 7 lines

void irq_complete_move(struct irq_cfg *cfg)

{
         __irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}

in arch/x86/kernel/apic/vector.c.

Thanks a lot, I stumbled over the same thing just after I sent the email as well and had been trying to see if I can quickly patch it up before I fall asleep :).

The code path above is used by the APIC vector move (irqbalance) logic, which explains why not everyone was seeing issues.

So with 633260fa1 applied, we never get out of moving state for our IRQ because orig_ax is always -1. That means we send an IPI to the cleanup vector on every single device interrupt, completely occupying the poor CPU that we moved the IRQ from.

I've confirmed that the patch below fixes the issue and will send a proper, more complete patch on top of mainline with fancy description and stable tag tomorrow.

Alex

arch/x86/entry/entry_64.S +0 -2
index e7434cd..a474e6e 100644
@@ -734,7 +734,6 @@ SYM_CODE_START_LOCAL(common_spurious)
        call    interrupt_entry
        UNWIND_HINT_REGS indirect=1
        movq    ORIG_RAX(%rdi), %rsi            /* get vector from stack */
-       movq    $-1, ORIG_RAX(%rdi)             /* no syscall to restart */
        call    smp_spurious_interrupt          /* rdi points to pt_regs */
        jmp     ret_from_intr
  SYM_CODE_END(common_spurious)
@@ -746,7 +745,6 @@ SYM_CODE_START_LOCAL(common_interrupt)
        call    interrupt_entry
        UNWIND_HINT_REGS indirect=1
        movq    ORIG_RAX(%rdi), %rsi            /* get vector from stack */
-       movq    $-1, ORIG_RAX(%rdi)             /* no syscall to restart */
        call    do_IRQ                          /* rdi points to pt_regs */
        /* 0(%rsp): old RSP */
  ret_from_intr:
arch/x86/kernel/apic/vector.c +0 -0
index 67768e5443..5b6f74e 100644

arch/x86/kernel/apic/vector.c static void __irq_complete_move(struct irq_cfg *cfg, · line 939

Quoted 7 lines
arch/x86/kernel/apic/vector.c +1 -1
@@ -934,7 +934,7 @@ static void __irq_complete_move(struct irq_cfg *cfg,
 unsigned vector)
 
  void irq_complete_move(struct irq_cfg *cfg)
  {
-       __irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
+       __irq_complete_move(cfg, get_irq_regs()->orig_ax);
  }

I think you need to also truncate the vector to 8-bits, since it now gets sign-extended when pushed into the orig_ax slot.

--
Brian Gerst

#77 lore raw

Re: [patch V9 21/39] x86/irq: Convey vector as argument and not in ptregs

In reply to Andy Lutomirski
Quoted 8 lines · Andy Lutomirski

How about this:

void irq_complete_move(struct irq_cfg *cfg)

{
        __irq_complete_move(cfg, ~get_irq_regs()->orig_ax);
}

in arch/x86/kernel/apic/vector.c.

Duh....

#78 lore raw

PATCH v9 22/39 x86/irq: Rework handle_irq() for 64bit

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

To consolidate the interrupt entry/exit code vs. the other exceptions make handle_irq() and inline and handle both 64bit and 32bit

Preparatory change to move irq stack switching for 64bit to C which allows to consolidate the entry exit handling by reusing the idtentry machinery both in ASM and C.

---
V9: Adopt to inlined run_cond_on_irqstack()

---
 arch/x86/include/asm/irq.h |    2 +-
 arch/x86/kernel/irq.c      |   11 ++++++++++-
 arch/x86/kernel/irq_32.c   |    2 +-
 3 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/include/asm/irq.h +1 -1
@@ -34,7 +34,7 @@ extern __visible void smp_kvm_posted_int
 extern void (*x86_platform_ipi_callback)(void);
 extern void native_init_IRQ(void);
 
-extern void handle_irq(struct irq_desc *desc, struct pt_regs *regs);
+extern void __handle_irq(struct irq_desc *desc, struct pt_regs *regs);
 
 extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);
 
arch/x86/kernel/irq.c +10 -1
@@ -13,6 +13,7 @@
 #include <linux/export.h>
 #include <linux/irq.h>
 
+#include <asm/irq_stack.h>
 #include <asm/apic.h>
 #include <asm/io_apic.h>
 #include <asm/irq.h>
@@ -221,6 +222,14 @@ u64 arch_irq_stat(void)
 	return sum;
 }
 
+static __always_inline void handle_irq(struct irq_desc *desc,
+				       struct pt_regs *regs)
+{
+	if (IS_ENABLED(CONFIG_X86_64))
+		run_on_irqstack_cond(desc->handle_irq, desc, regs);
+	else
+		__handle_irq(desc, regs);
+}
 
 /*
  * do_IRQ handles all normal device IRQ's (the special
@@ -246,7 +255,7 @@ u64 arch_irq_stat(void)
 	desc = __this_cpu_read(vector_irq[vector]);
 	if (likely(!IS_ERR_OR_NULL(desc))) {
 		if (IS_ENABLED(CONFIG_X86_32))
-			handle_irq(desc, regs);
+			__handle_irq(desc, regs);
 		else
 			generic_handle_irq_desc(desc);
 	} else {
arch/x86/kernel/irq_32.c +1 -1
@@ -148,7 +148,7 @@ void do_softirq_own_stack(void)
 	call_on_stack(__do_softirq, isp);
 }
 
-void handle_irq(struct irq_desc *desc, struct pt_regs *regs)
+void __handle_irq(struct irq_desc *desc, struct pt_regs *regs)
 {
 	int overflow = check_stack_overflow();
 
#79 lore raw

[tip: x86/entry] x86/irq: Rework handle_irq() for 64-bit

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     9fe3411558d31f9489cd58400b2f508a98dc13a5
Gitweb:        https://git.kernel.org/tip/9fe3411558d31f9489cd58400b2f508a98dc13a5
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:35 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/irq: Rework handle_irq() for 64-bit

To consolidate the interrupt entry/exit code vs. the other exceptions make handle_irq() and inline and handle both 64-bit and 32-bit mode.

Preparatory change to move irq stack switching for 64-bit to C which allows to consolidate the entry exit handling by reusing the idtentry machinery both in ASM and C.

---
 arch/x86/include/asm/irq.h |  2 +-
 arch/x86/kernel/irq.c      | 11 ++++++++++-
 arch/x86/kernel/irq_32.c   |  2 +-
 3 files changed, 12 insertions(+), 3 deletions(-)
arch/x86/include/asm/irq.h +1 -1
index 74690a3..67aa1e2 100644
@@ -34,7 +34,7 @@ extern __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs);
 extern void (*x86_platform_ipi_callback)(void);
 extern void native_init_IRQ(void);
 
-extern void handle_irq(struct irq_desc *desc, struct pt_regs *regs);
+extern void __handle_irq(struct irq_desc *desc, struct pt_regs *regs);
 
 extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);
 
arch/x86/kernel/irq.c +10 -1
index c766936..5495ea4 100644
@@ -13,6 +13,7 @@
 #include <linux/export.h>
 #include <linux/irq.h>
 
+#include <asm/irq_stack.h>
 #include <asm/apic.h>
 #include <asm/io_apic.h>
 #include <asm/irq.h>
@@ -221,6 +222,14 @@ u64 arch_irq_stat(void)
 	return sum;
 }
 
+static __always_inline void handle_irq(struct irq_desc *desc,
+				       struct pt_regs *regs)
+{
+	if (IS_ENABLED(CONFIG_X86_64))
+		run_on_irqstack_cond(desc->handle_irq, desc, regs);
+	else
+		__handle_irq(desc, regs);
+}
 
 /*
  * do_IRQ handles all normal device IRQ's (the special
@@ -246,7 +255,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector)
 	desc = __this_cpu_read(vector_irq[vector]);
 	if (likely(!IS_ERR_OR_NULL(desc))) {
 		if (IS_ENABLED(CONFIG_X86_32))
-			handle_irq(desc, regs);
+			__handle_irq(desc, regs);
 		else
 			generic_handle_irq_desc(desc);
 	} else {
arch/x86/kernel/irq_32.c +1 -1
index a759ca9..0b79efc 100644
@@ -148,7 +148,7 @@ void do_softirq_own_stack(void)
 	call_on_stack(__do_softirq, isp);
 }
 
-void handle_irq(struct irq_desc *desc, struct pt_regs *regs)
+void __handle_irq(struct irq_desc *desc, struct pt_regs *regs)
 {
 	int overflow = check_stack_overflow();
 
#80 lore raw

PATCH v9 23/39 x86/entry: Add IRQENTRY_IRQ macro

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Provide a seperate IDTENTRY macro for device interrupts. Similar to IDTENTRY_ERRORCODE with the addition of invoking irq_enter/exit_rcu() and providing the errorcode as a 'u8' argument to the C function, which truncates the sign extended vector number.

---
V9: Use cond RCU handling

---
 arch/x86/entry/entry_32.S       |   14 +++++++++++
 arch/x86/entry/entry_64.S       |   14 +++++++++++
 arch/x86/include/asm/idtentry.h |   48 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)
arch/x86/entry/entry_32.S +14 -0
@@ -751,6 +751,20 @@ SYM_CODE_START(\asmsym)
 SYM_CODE_END(\asmsym)
 .endm
 
+.macro idtentry_irq vector cfunc
+	.p2align CONFIG_X86_L1_CACHE_SHIFT
+SYM_CODE_START_LOCAL(asm_\cfunc)
+	ASM_CLAC
+	SAVE_ALL switch_stacks=1
+	ENCODE_FRAME_POINTER
+	movl	%esp, %eax
+	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
+	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
+	call	\cfunc
+	jmp	handle_exception_return
+SYM_CODE_END(asm_\cfunc)
+.endm
+
 /*
  * Include the defines which emit the idt entries which are shared
  * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +14 -0
@@ -528,6 +528,20 @@ SYM_CODE_END(\asmsym)
 .endm
 
 /*
+ * Interrupt entry/exit.
+ *
+ + The interrupt stubs push (vector) onto the stack, which is the error_code
+ * position of idtentry exceptions, and jump to one of the two idtentry points
+ * (common/spurious).
+ *
+ * common_interrupt is a hotpath, align it to a cache line
+ */
+.macro idtentry_irq vector cfunc
+	.p2align CONFIG_X86_L1_CACHE_SHIFT
+	idtentry \vector asm_\cfunc \cfunc has_error_code=1
+.endm
+
+/*
  * MCE and DB exceptions
  */
 #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +48 -0

60 lines not on this page · raw message

#81 lore raw

[tip: x86/entry] x86/entry: Add IRQENTRY_IRQ macro

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     f7376b1a72bb4bcfe2212150e3cf79f21cf88912
Gitweb:        https://git.kernel.org/tip/f7376b1a72bb4bcfe2212150e3cf79f21cf88912
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:36 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Add IRQENTRY_IRQ macro

Provide a seperate IDTENTRY macro for device interrupts. Similar to IDTENTRY_ERRORCODE with the addition of invoking irq_enter/exit_rcu() and providing the errorcode as a 'u8' argument to the C function, which truncates the sign extended vector number.

---
 arch/x86/entry/entry_32.S       | 14 +++++++++-
 arch/x86/entry/entry_64.S       | 14 +++++++++-
 arch/x86/include/asm/idtentry.h | 48 ++++++++++++++++++++++++++++++++-
 3 files changed, 76 insertions(+)
arch/x86/entry/entry_32.S +14 -0
index 53a6447..6930ec1 100644
@@ -751,6 +751,20 @@ SYM_CODE_START(\asmsym)
 SYM_CODE_END(\asmsym)
 .endm
 
+.macro idtentry_irq vector cfunc
+	.p2align CONFIG_X86_L1_CACHE_SHIFT
+SYM_CODE_START_LOCAL(asm_\cfunc)
+	ASM_CLAC
+	SAVE_ALL switch_stacks=1
+	ENCODE_FRAME_POINTER
+	movl	%esp, %eax
+	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
+	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
+	call	\cfunc
+	jmp	handle_exception_return
+SYM_CODE_END(asm_\cfunc)
+.endm
+
 /*
  * Include the defines which emit the idt entries which are shared
  * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +14 -0
index e7434cd..9162a07 100644
@@ -528,6 +528,20 @@ SYM_CODE_END(\asmsym)
 .endm
 
 /*
+ * Interrupt entry/exit.
+ *
+ + The interrupt stubs push (vector) onto the stack, which is the error_code
+ * position of idtentry exceptions, and jump to one of the two idtentry points
+ * (common/spurious).
+ *
+ * common_interrupt is a hotpath, align it to a cache line
+ */
+.macro idtentry_irq vector cfunc
+	.p2align CONFIG_X86_L1_CACHE_SHIFT
+	idtentry \vector asm_\cfunc \cfunc has_error_code=1
+.endm
+
+/*
  * MCE and DB exceptions
  */
 #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +48 -0

60 lines not on this page · raw message

#82 lore raw

PATCH v9 24/39 x86/entry: Use idtentry for interrupts

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Replace the extra interrupt handling code and reuse the existing idtentry machinery. This moves the irq stack switching on 64 bit from ASM to C code; 32bit already does the stack switching in C.

---
 arch/x86/entry/entry_32.S       |   31 -------------------------------
 arch/x86/entry/entry_64.S       |   31 +++----------------------------
 arch/x86/include/asm/hw_irq.h   |    1 -
 arch/x86/include/asm/idtentry.h |   10 ++++++++--
 arch/x86/include/asm/traps.h    |    1 -
 arch/x86/kernel/apic/apic.c     |   23 ++++++++---------------
 arch/x86/kernel/apic/msi.c      |    3 ++-
 arch/x86/kernel/irq.c           |   27 +++++++--------------------
 8 files changed, 28 insertions(+), 99 deletions(-)
arch/x86/entry/entry_32.S +0 -31
@@ -1229,37 +1229,6 @@ SYM_FUNC_END(entry_INT80_32)
 #endif
 .endm
 
-#ifdef CONFIG_X86_LOCAL_APIC
-SYM_CODE_START_LOCAL(common_spurious)
-	ASM_CLAC
-	SAVE_ALL switch_stacks=1
-	ENCODE_FRAME_POINTER
-	TRACE_IRQS_OFF
-	movl	%esp, %eax
-	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
-	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
-	call	smp_spurious_interrupt
-	jmp	ret_from_intr
-SYM_CODE_END(common_spurious)
-#endif
-
-/*
- * the CPU automatically disables interrupts when executing an IRQ vector,
- * so IRQ-flags tracing has to follow that:
- */
-	.p2align CONFIG_X86_L1_CACHE_SHIFT
-SYM_CODE_START_LOCAL(common_interrupt)
-	ASM_CLAC
-	SAVE_ALL switch_stacks=1
-	ENCODE_FRAME_POINTER
-	TRACE_IRQS_OFF
-	movl	%esp, %eax
-	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
-	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
-	call	do_IRQ
-	jmp	ret_from_intr
-SYM_CODE_END(common_interrupt)
-
 #define BUILD_INTERRUPT3(name, nr, fn)			\
 SYM_FUNC_START(name)					\
 	ASM_CLAC;					\
arch/x86/entry/entry_64.S +3 -28
@@ -737,32 +737,7 @@ SYM_CODE_START(interrupt_entry)
 SYM_CODE_END(interrupt_entry)
 _ASM_NOKPROBE(interrupt_entry)
 
-
-/* Interrupt entry/exit. */
-
-/*
- * The interrupt stubs push vector onto the stack and
- * then jump to common_spurious/interrupt.
- */
-SYM_CODE_START_LOCAL(common_spurious)
-	call	interrupt_entry
-	UNWIND_HINT_REGS indirect=1
-	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
-	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
-	call	smp_spurious_interrupt		/* rdi points to pt_regs */
-	jmp	ret_from_intr
-SYM_CODE_END(common_spurious)
-_ASM_NOKPROBE(common_spurious)
-
-/* common_interrupt is a hotpath. Align it */
-	.p2align CONFIG_X86_L1_CACHE_SHIFT
-SYM_CODE_START_LOCAL(common_interrupt)
-	call	interrupt_entry
-	UNWIND_HINT_REGS indirect=1
-	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
-	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
-	call	do_IRQ				/* rdi points to pt_regs */
-	/* 0(%rsp): old RSP */
+SYM_CODE_START_LOCAL(common_interrupt_return)
 ret_from_intr:
 	DISABLE_INTERRUPTS(CLBR_ANY)
 	TRACE_IRQS_OFF
@@ -945,8 +920,8 @@ SYM_INNER_LABEL(native_irq_return_iret,
 	 */
 	jmp	native_irq_return_iret
 #endif
-SYM_CODE_END(common_interrupt)
-_ASM_NOKPROBE(common_interrupt)
+SYM_CODE_END(common_interrupt_return)
+_ASM_NOKPROBE(common_interrupt_return)
 
 /*
  * APIC interrupts.
arch/x86/include/asm/hw_irq.h +0 -1
@@ -38,7 +38,6 @@ extern asmlinkage void error_interrupt(v
 extern asmlinkage void irq_work_interrupt(void);
 extern asmlinkage void uv_bau_message_intr1(void);
 
-extern asmlinkage void spurious_interrupt(void);
 extern asmlinkage void spurious_apic_interrupt(void);
 extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +8 -2
@@ -395,7 +395,7 @@ SYM_CODE_START(irq_entries_start)
     .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
 	UNWIND_HINT_IRET_REGS
 	.byte	0x6a, vector
-	jmp	common_interrupt
+	jmp	asm_common_interrupt
 	.nops (pos + 8) - .
     pos=pos+8
     vector=vector+1
@@ -410,7 +410,7 @@ SYM_CODE_START(spurious_entries_start)
     .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR)
 	UNWIND_HINT_IRET_REGS
 	.byte	0x6a, vector
-	jmp	common_spurious
+	jmp	asm_spurious_interrupt
 	.nops (pos + 8) - .
     pos=pos+8
     vector=vector+1
@@ -480,6 +480,12 @@ DECLARE_IDTENTRY_DF(X86_TRAP_DF,	exc_dou
 DECLARE_IDTENTRY(X86_TRAP_OTHER,	exc_xen_hypervisor_callback);
 #endif
 
+/* Device interrupts common/spurious */
+DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER,	common_interrupt);
+#ifdef CONFIG_X86_LOCAL_APIC
+DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER,	spurious_interrupt);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/traps.h +0 -1
@@ -43,7 +43,6 @@ asmlinkage void smp_deferred_error_inter
 void smp_apic_timer_interrupt(struct pt_regs *regs);
 void smp_error_interrupt(struct pt_regs *regs);
 void smp_spurious_apic_interrupt(struct pt_regs *regs);
-void smp_spurious_interrupt(struct pt_regs *regs, unsigned long vector);
 asmlinkage void smp_irq_move_cleanup_interrupt(void);
 
 #ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +8 -15
@@ -2154,9 +2154,9 @@ void __init register_lapic_address(unsig
  */
 
 /**
- * smp_spurious_interrupt - Catch all for interrupts raised on unused vectors
+ * spurious_interrupt - Catch all for interrupts raised on unused vectors
  * @regs:	Pointer to pt_regs on stack
- * @error_code:	The vector number is in the lower 8 bits
+ * @vector:	The vector number
  *
  * This is invoked from ASM entry code to catch all interrupts which
  * trigger on an entry which is routed to the common_spurious idtentry
@@ -2164,18 +2164,10 @@ void __init register_lapic_address(unsig
  *
  * Also called from smp_spurious_apic_interrupt().
  */
-__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs,
-						  unsigned long vector)
+DEFINE_IDTENTRY_IRQ(spurious_interrupt)
 {
 	u32 v;
 
-	entering_irq();
-	/*
-	 * The push in the entry ASM code which stores the vector number on
-	 * the stack in the error code slot is sign expanding. Just use the
-	 * lower 8 bits.
-	 */
-	vector &= 0xFF;
 	trace_spurious_apic_entry(vector);
 
 	inc_irq_stat(irq_spurious_count);
@@ -2196,21 +2188,22 @@ void __init register_lapic_address(unsig
 	 */
 	v = apic_read(APIC_ISR + ((vector & ~0x1f) >> 1));
 	if (v & (1 << (vector & 0x1f))) {
-		pr_info("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n",
+		pr_info("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n",
 			vector, smp_processor_id());
 		ack_APIC_irq();
 	} else {
-		pr_info("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n",
+		pr_info("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n",
 			vector, smp_processor_id());
 	}
 out:
 	trace_spurious_apic_exit(vector);
-	exiting_irq();
 }
 
 __visible void smp_spurious_apic_interrupt(struct pt_regs *regs)
 {
-	smp_spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+	entering_irq();
+	__spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+	exiting_irq();
 }
 
 /*
arch/x86/kernel/apic/msi.c +2 -1
@@ -115,7 +115,8 @@ msi_set_affinity(struct irq_data *irqd,
 	 * denote it as spurious which is no harm as this is a rare event
 	 * and interrupt handlers have to cope with spurious interrupts
 	 * anyway. If the vector is unused, then it is marked so it won't
-	 * trigger the 'No irq handler for vector' warning in do_IRQ().
+	 * trigger the 'No irq handler for vector' warning in
+	 * common_interrupt().
 	 *
 	 * This requires to hold vector lock to prevent concurrent updates to
 	 * the affected vector.
arch/x86/kernel/irq.c +7 -20
@@ -20,6 +20,7 @@
 #include <asm/mce.h>
 #include <asm/hw_irq.h>
 #include <asm/desc.h>
+#include <asm/traps.h>
 
 #define CREATE_TRACE_POINTS
 #include <asm/trace/irq_vectors.h>
@@ -232,37 +233,25 @@ static __always_inline void handle_irq(s
 }
 
 /*
- * do_IRQ handles all normal device IRQ's (the special
- * SMP cross-CPU interrupts have their own specific
- * handlers).
+ * common_interrupt() handles all normal device IRQ's (the special SMP
+ * cross-CPU interrupts have their own entry points).
  */
-__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector)
+DEFINE_IDTENTRY_IRQ(common_interrupt)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 	struct irq_desc *desc;
 
-	entering_irq();
-	/*
-	 * The push in the entry ASM code which stores the vector number on
-	 * the stack in the error code slot is sign expanding. Just use the
-	 * lower 8 bits.
-	 */
-	vector &= 0xFF;
-
-	/* entering_irq() tells RCU that we're not quiescent.  Check it. */
+	/* entry code tells RCU that we're not quiescent.  Check it. */
 	RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
 
 	desc = __this_cpu_read(vector_irq[vector]);
 	if (likely(!IS_ERR_OR_NULL(desc))) {
-		if (IS_ENABLED(CONFIG_X86_32))
-			__handle_irq(desc, regs);
-		else
-			generic_handle_irq_desc(desc);
+		handle_irq(desc, regs);
 	} else {
 		ack_APIC_irq();
 
 		if (desc == VECTOR_UNUSED) {
-			pr_emerg_ratelimited("%s: %d.%lu No irq handler for vector\n",
+			pr_emerg_ratelimited("%s: %d.%u No irq handler for vector\n",
 					     __func__, smp_processor_id(),
 					     vector);
 		} else {
@@ -270,8 +259,6 @@ static __always_inline void handle_irq(s
 		}
 	}
 
-	exiting_irq();
-
 	set_irq_regs(old_regs);
 }
 
#83 lore raw

[tip: x86/entry] x86/entry: Use idtentry for interrupts

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     e1177738020f5ad104b5a5c31c78b1b4252deaca
Gitweb:        https://git.kernel.org/tip/e1177738020f5ad104b5a5c31c78b1b4252deaca
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:37 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Use idtentry for interrupts

Replace the extra interrupt handling code and reuse the existing idtentry machinery. This moves the irq stack switching on 64-bit from ASM to C code; 32-bit already does the stack switching in C.

---
 arch/x86/entry/entry_32.S       | 31 -------------------------------
 arch/x86/entry/entry_64.S       | 31 +++----------------------------
 arch/x86/include/asm/hw_irq.h   |  1 -
 arch/x86/include/asm/idtentry.h | 10 ++++++++--
 arch/x86/include/asm/traps.h    |  1 -
 arch/x86/kernel/apic/apic.c     | 23 ++++++++---------------
 arch/x86/kernel/apic/msi.c      |  3 ++-
 arch/x86/kernel/irq.c           | 27 +++++++--------------------
 8 files changed, 28 insertions(+), 99 deletions(-)
arch/x86/entry/entry_32.S +0 -31
index 6930ec1..d86e056 100644
@@ -1229,37 +1229,6 @@ SYM_FUNC_END(entry_INT80_32)
 #endif
 .endm
 
-#ifdef CONFIG_X86_LOCAL_APIC
-SYM_CODE_START_LOCAL(common_spurious)
-	ASM_CLAC
-	SAVE_ALL switch_stacks=1
-	ENCODE_FRAME_POINTER
-	TRACE_IRQS_OFF
-	movl	%esp, %eax
-	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
-	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
-	call	smp_spurious_interrupt
-	jmp	ret_from_intr
-SYM_CODE_END(common_spurious)
-#endif
-
-/*
- * the CPU automatically disables interrupts when executing an IRQ vector,
- * so IRQ-flags tracing has to follow that:
- */
-	.p2align CONFIG_X86_L1_CACHE_SHIFT
-SYM_CODE_START_LOCAL(common_interrupt)
-	ASM_CLAC
-	SAVE_ALL switch_stacks=1
-	ENCODE_FRAME_POINTER
-	TRACE_IRQS_OFF
-	movl	%esp, %eax
-	movl	PT_ORIG_EAX(%esp), %edx		/* get the vector from stack */
-	movl	$-1, PT_ORIG_EAX(%esp)		/* no syscall to restart */
-	call	do_IRQ
-	jmp	ret_from_intr
-SYM_CODE_END(common_interrupt)
-
 #define BUILD_INTERRUPT3(name, nr, fn)			\
 SYM_FUNC_START(name)					\
 	ASM_CLAC;					\
arch/x86/entry/entry_64.S +3 -28
index 9162a07..e54bcd3 100644
@@ -737,32 +737,7 @@ SYM_CODE_START(interrupt_entry)
 SYM_CODE_END(interrupt_entry)
 _ASM_NOKPROBE(interrupt_entry)
 
-
-/* Interrupt entry/exit. */
-
-/*
- * The interrupt stubs push vector onto the stack and
- * then jump to common_spurious/interrupt.
- */
-SYM_CODE_START_LOCAL(common_spurious)
-	call	interrupt_entry
-	UNWIND_HINT_REGS indirect=1
-	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
-	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
-	call	smp_spurious_interrupt		/* rdi points to pt_regs */
-	jmp	ret_from_intr
-SYM_CODE_END(common_spurious)
-_ASM_NOKPROBE(common_spurious)
-
-/* common_interrupt is a hotpath. Align it */
-	.p2align CONFIG_X86_L1_CACHE_SHIFT
-SYM_CODE_START_LOCAL(common_interrupt)
-	call	interrupt_entry
-	UNWIND_HINT_REGS indirect=1
-	movq	ORIG_RAX(%rdi), %rsi		/* get vector from stack */
-	movq	$-1, ORIG_RAX(%rdi)		/* no syscall to restart */
-	call	do_IRQ				/* rdi points to pt_regs */
-	/* 0(%rsp): old RSP */
+SYM_CODE_START_LOCAL(common_interrupt_return)
 ret_from_intr:
 	DISABLE_INTERRUPTS(CLBR_ANY)
 	TRACE_IRQS_OFF
@@ -945,8 +920,8 @@ native_irq_return_ldt:
 	 */
 	jmp	native_irq_return_iret
 #endif
-SYM_CODE_END(common_interrupt)
-_ASM_NOKPROBE(common_interrupt)
+SYM_CODE_END(common_interrupt_return)
+_ASM_NOKPROBE(common_interrupt_return)
 
 /*
  * APIC interrupts.
arch/x86/include/asm/hw_irq.h +0 -1
index 0ffe807..3213d36 100644
@@ -38,7 +38,6 @@ extern asmlinkage void error_interrupt(void);
 extern asmlinkage void irq_work_interrupt(void);
 extern asmlinkage void uv_bau_message_intr1(void);
 
-extern asmlinkage void spurious_interrupt(void);
 extern asmlinkage void spurious_apic_interrupt(void);
 extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +8 -2
index 6e00055..32035d1 100644
@@ -395,7 +395,7 @@ SYM_CODE_START(irq_entries_start)
     .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR)
 	UNWIND_HINT_IRET_REGS
 	.byte	0x6a, vector
-	jmp	common_interrupt
+	jmp	asm_common_interrupt
 	nop
 	/* Ensure that the above is 8 bytes max */
 	. = pos + 8
@@ -412,7 +412,7 @@ SYM_CODE_START(spurious_entries_start)
     .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR)
 	UNWIND_HINT_IRET_REGS
 	.byte	0x6a, vector
-	jmp	common_spurious
+	jmp	asm_spurious_interrupt
 	nop
 	/* Ensure that the above is 8 bytes max */
 	. = pos + 8
@@ -484,6 +484,12 @@ DECLARE_IDTENTRY_DF(X86_TRAP_DF,	exc_double_fault);
 DECLARE_IDTENTRY(X86_TRAP_OTHER,	exc_xen_hypervisor_callback);
 #endif
 
+/* Device interrupts common/spurious */
+DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER,	common_interrupt);
+#ifdef CONFIG_X86_LOCAL_APIC
+DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER,	spurious_interrupt);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/traps.h +0 -1
index 32b2bec..97e6945 100644
@@ -43,7 +43,6 @@ asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);
 void smp_apic_timer_interrupt(struct pt_regs *regs);
 void smp_error_interrupt(struct pt_regs *regs);
 void smp_spurious_apic_interrupt(struct pt_regs *regs);
-void smp_spurious_interrupt(struct pt_regs *regs, unsigned long vector);
 asmlinkage void smp_irq_move_cleanup_interrupt(void);
 
 #ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +8 -15
index a4218a3..66c3cfc 100644
@@ -2154,9 +2154,9 @@ void __init register_lapic_address(unsigned long address)
  */
 
 /**
- * smp_spurious_interrupt - Catch all for interrupts raised on unused vectors
+ * spurious_interrupt - Catch all for interrupts raised on unused vectors
  * @regs:	Pointer to pt_regs on stack
- * @error_code:	The vector number is in the lower 8 bits
+ * @vector:	The vector number
  *
  * This is invoked from ASM entry code to catch all interrupts which
  * trigger on an entry which is routed to the common_spurious idtentry
@@ -2164,18 +2164,10 @@ void __init register_lapic_address(unsigned long address)
  *
  * Also called from smp_spurious_apic_interrupt().
  */
-__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs,
-						  unsigned long vector)
+DEFINE_IDTENTRY_IRQ(spurious_interrupt)
 {
 	u32 v;
 
-	entering_irq();
-	/*
-	 * The push in the entry ASM code which stores the vector number on
-	 * the stack in the error code slot is sign expanding. Just use the
-	 * lower 8 bits.
-	 */
-	vector &= 0xFF;
 	trace_spurious_apic_entry(vector);
 
 	inc_irq_stat(irq_spurious_count);
@@ -2196,21 +2188,22 @@ __visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs,
 	 */
 	v = apic_read(APIC_ISR + ((vector & ~0x1f) >> 1));
 	if (v & (1 << (vector & 0x1f))) {
-		pr_info("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n",
+		pr_info("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n",
 			vector, smp_processor_id());
 		ack_APIC_irq();
 	} else {
-		pr_info("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n",
+		pr_info("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n",
 			vector, smp_processor_id());
 	}
 out:
 	trace_spurious_apic_exit(vector);
-	exiting_irq();
 }
 
 __visible void smp_spurious_apic_interrupt(struct pt_regs *regs)
 {
-	smp_spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+	entering_irq();
+	__spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
+	exiting_irq();
 }
 
 /*
arch/x86/kernel/apic/msi.c +2 -1
index 159bd0c..5cbaca5 100644
@@ -115,7 +115,8 @@ msi_set_affinity(struct irq_data *irqd, const struct cpumask *mask, bool force)
 	 * denote it as spurious which is no harm as this is a rare event
 	 * and interrupt handlers have to cope with spurious interrupts
 	 * anyway. If the vector is unused, then it is marked so it won't
-	 * trigger the 'No irq handler for vector' warning in do_IRQ().
+	 * trigger the 'No irq handler for vector' warning in
+	 * common_interrupt().
 	 *
 	 * This requires to hold vector lock to prevent concurrent updates to
 	 * the affected vector.
arch/x86/kernel/irq.c +7 -20
index 5495ea4..c449b84 100644
@@ -20,6 +20,7 @@
 #include <asm/mce.h>
 #include <asm/hw_irq.h>
 #include <asm/desc.h>
+#include <asm/traps.h>
 
 #define CREATE_TRACE_POINTS
 #include <asm/trace/irq_vectors.h>
@@ -232,37 +233,25 @@ static __always_inline void handle_irq(struct irq_desc *desc,
 }
 
 /*
- * do_IRQ handles all normal device IRQ's (the special
- * SMP cross-CPU interrupts have their own specific
- * handlers).
+ * common_interrupt() handles all normal device IRQ's (the special SMP
+ * cross-CPU interrupts have their own entry points).
  */
-__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector)
+DEFINE_IDTENTRY_IRQ(common_interrupt)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 	struct irq_desc *desc;
 
-	entering_irq();
-	/*
-	 * The push in the entry ASM code which stores the vector number on
-	 * the stack in the error code slot is sign expanding. Just use the
-	 * lower 8 bits.
-	 */
-	vector &= 0xFF;
-
-	/* entering_irq() tells RCU that we're not quiescent.  Check it. */
+	/* entry code tells RCU that we're not quiescent.  Check it. */
 	RCU_LOCKDEP_WARN(!rcu_is_watching(), "IRQ failed to wake up RCU");
 
 	desc = __this_cpu_read(vector_irq[vector]);
 	if (likely(!IS_ERR_OR_NULL(desc))) {
-		if (IS_ENABLED(CONFIG_X86_32))
-			__handle_irq(desc, regs);
-		else
-			generic_handle_irq_desc(desc);
+		handle_irq(desc, regs);
 	} else {
 		ack_APIC_irq();
 
 		if (desc == VECTOR_UNUSED) {
-			pr_emerg_ratelimited("%s: %d.%lu No irq handler for vector\n",
+			pr_emerg_ratelimited("%s: %d.%u No irq handler for vector\n",
 					     __func__, smp_processor_id(),
 					     vector);
 		} else {
@@ -270,8 +259,6 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector)
 		}
 	}
 
-	exiting_irq();
-
 	set_irq_regs(old_regs);
 }
 
#84 lore raw

PATCH v9 25/39 x86/entry: Provide IDTENTRY_SYSVEC

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Provide IDTENTRY variants for system vectors to consolidate the different mechanisms to emit the ASM stubs for 32 an 64 bit.

On 64bit this also moves the stack switching from ASM to C code. 32bit will excute the system vectors w/o stack switching as before.

The simple variant is meant for "empty" system vectors like scheduler IPI and KVM posted interrupt vectors. These do not need the full glory of irq enter/exit handling with softirq processing and more.

---
V9: Use cond. RCU handling

---
 arch/x86/entry/entry_32.S       |    4 ++
 arch/x86/entry/entry_64.S       |    8 ++++
 arch/x86/include/asm/idtentry.h |   79 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 91 insertions(+)
arch/x86/entry/entry_32.S +4 -0
@@ -765,6 +765,10 @@ SYM_CODE_START_LOCAL(asm_\cfunc)
 SYM_CODE_END(asm_\cfunc)
 .endm
 
+.macro idtentry_sysvec vector cfunc
+	idtentry \vector asm_\cfunc \cfunc has_error_code=0
+.endm
+
 /*
  * Include the defines which emit the idt entries which are shared
  * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +8 -0
@@ -542,6 +542,14 @@ SYM_CODE_END(\asmsym)
 .endm
 
 /*
+ * System vectors which invoke their handlers directly and are not
+ * going through the regular common device interrupt handling code.
+ */
+.macro idtentry_sysvec vector cfunc
+	idtentry \vector asm_\cfunc \cfunc has_error_code=0
+.endm
+
+/*
  * MCE and DB exceptions
  */
 #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +79 -0

97 lines not on this page · raw message

#85 lore raw

[tip: x86/entry] x86/entry: Provide IDTENTRY_SYSVEC

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     8c3d969a5352c3e68a42311852e20ee7530492c6
Gitweb:        https://git.kernel.org/tip/8c3d969a5352c3e68a42311852e20ee7530492c6
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:38 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Provide IDTENTRY_SYSVEC

Provide IDTENTRY variants for system vectors to consolidate the different mechanisms to emit the ASM stubs for 32- and 64-bit.

On 64-bit this also moves the stack switching from ASM to C code. 32-bit will excute the system vectors w/o stack switching as before.

The simple variant is meant for "empty" system vectors like scheduler IPI and KVM posted interrupt vectors. These do not need the full glory of irq enter/exit handling with softirq processing and more.

---
 arch/x86/entry/entry_32.S       |  4 ++-
 arch/x86/entry/entry_64.S       |  8 +++-
 arch/x86/include/asm/idtentry.h | 79 ++++++++++++++++++++++++++++++++-
 3 files changed, 91 insertions(+)
arch/x86/entry/entry_32.S +4 -0
index d86e056..e59ad4b 100644
@@ -765,6 +765,10 @@ SYM_CODE_START_LOCAL(asm_\cfunc)
 SYM_CODE_END(asm_\cfunc)
 .endm
 
+.macro idtentry_sysvec vector cfunc
+	idtentry \vector asm_\cfunc \cfunc has_error_code=0
+.endm
+
 /*
  * Include the defines which emit the idt entries which are shared
  * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +8 -0
index e54bcd3..9b7183d 100644
@@ -542,6 +542,14 @@ SYM_CODE_END(\asmsym)
 .endm
 
 /*
+ * System vectors which invoke their handlers directly and are not
+ * going through the regular common device interrupt handling code.
+ */
+.macro idtentry_sysvec vector cfunc
+	idtentry \vector asm_\cfunc \cfunc has_error_code=0
+.endm
+
+/*
  * MCE and DB exceptions
  */
 #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +79 -0

97 lines not on this page · raw message

#86 lore raw

PATCH v9 26/39 x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Convert APIC interrupts to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes

No functional change.

---
 arch/x86/entry/entry_64.S         |    6 ------
 arch/x86/include/asm/entry_arch.h |    5 -----
 arch/x86/include/asm/hw_irq.h     |    4 ----
 arch/x86/include/asm/idtentry.h   |    8 ++++++++
 arch/x86/include/asm/irq.h        |    1 -
 arch/x86/include/asm/traps.h      |    3 ---
 arch/x86/kernel/apic/apic.c       |   23 +++++------------------
 arch/x86/kernel/idt.c             |    8 ++++----
 arch/x86/kernel/irq.c             |    5 ++---
 9 files changed, 19 insertions(+), 44 deletions(-)
arch/x86/entry/entry_64.S +0 -6
@@ -965,9 +965,6 @@ apicinterrupt3 REBOOT_VECTOR			reboot_in
 apicinterrupt3 UV_BAU_MESSAGE			uv_bau_message_intr1		uv_bau_message_interrupt
 #endif
 
-apicinterrupt LOCAL_TIMER_VECTOR		apic_timer_interrupt		smp_apic_timer_interrupt
-apicinterrupt X86_PLATFORM_IPI_VECTOR		x86_platform_ipi		smp_x86_platform_ipi
-
 #ifdef CONFIG_HAVE_KVM
 apicinterrupt3 POSTED_INTR_VECTOR		kvm_posted_intr_ipi		smp_kvm_posted_intr_ipi
 apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR	kvm_posted_intr_wakeup_ipi	smp_kvm_posted_intr_wakeup_ipi
@@ -992,9 +989,6 @@ apicinterrupt CALL_FUNCTION_VECTOR		call
 apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
 
-apicinterrupt ERROR_APIC_VECTOR			error_interrupt			smp_error_interrupt
-apicinterrupt SPURIOUS_APIC_VECTOR		spurious_apic_interrupt		smp_spurious_apic_interrupt
-
 #ifdef CONFIG_IRQ_WORK
 apicinterrupt IRQ_WORK_VECTOR			irq_work_interrupt		smp_irq_work_interrupt
 #endif
arch/x86/include/asm/entry_arch.h +0 -5
@@ -33,11 +33,6 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_i
  */
 #ifdef CONFIG_X86_LOCAL_APIC
 
-BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR)
-BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR)
-BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR)
-BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR)
-
 #ifdef CONFIG_IRQ_WORK
 BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR)
 #endif
arch/x86/include/asm/hw_irq.h +0 -4
@@ -29,16 +29,12 @@
 #include <asm/sections.h>
 
 /* Interrupt handlers registered during init_IRQ */
-extern asmlinkage void apic_timer_interrupt(void);
-extern asmlinkage void x86_platform_ipi(void);
 extern asmlinkage void kvm_posted_intr_ipi(void);
 extern asmlinkage void kvm_posted_intr_wakeup_ipi(void);
 extern asmlinkage void kvm_posted_intr_nested_ipi(void);
-extern asmlinkage void error_interrupt(void);
 extern asmlinkage void irq_work_interrupt(void);
 extern asmlinkage void uv_bau_message_intr1(void);
 
-extern asmlinkage void spurious_apic_interrupt(void);
 extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
 
arch/x86/include/asm/idtentry.h +8 -0
@@ -565,6 +565,14 @@ DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER,	com
 DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER,	spurious_interrupt);
 #endif
 
+/* System vector entry points */
+#ifdef CONFIG_X86_LOCAL_APIC
+DECLARE_IDTENTRY_SYSVEC(ERROR_APIC_VECTOR,		sysvec_error_interrupt);
+DECLARE_IDTENTRY_SYSVEC(SPURIOUS_APIC_VECTOR,		sysvec_spurious_apic_interrupt);
+DECLARE_IDTENTRY_SYSVEC(LOCAL_TIMER_VECTOR,		sysvec_apic_timer_interrupt);
+DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR,	sysvec_x86_platform_ipi);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/irq.h +0 -1
@@ -46,7 +46,6 @@ extern void __init init_IRQ(void);
 void arch_trigger_cpumask_backtrace(const struct cpumask *mask,
 				    bool exclude_self);
 
-extern __visible void smp_x86_platform_ipi(struct pt_regs *regs);
 #define arch_trigger_cpumask_backtrace arch_trigger_cpumask_backtrace
 #endif
 
arch/x86/include/asm/traps.h +0 -3
@@ -40,9 +40,6 @@ asmlinkage void smp_threshold_interrupt(
 asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);
 #endif
 
-void smp_apic_timer_interrupt(struct pt_regs *regs);
-void smp_error_interrupt(struct pt_regs *regs);
-void smp_spurious_apic_interrupt(struct pt_regs *regs);
 asmlinkage void smp_irq_move_cleanup_interrupt(void);
 
 #ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +5 -18
@@ -1121,23 +1121,14 @@ static void local_apic_timer_interrupt(v
  * [ if a single-CPU system runs an SMP kernel then we call the local
  *   interrupt as well. Thus we cannot inline the local irq ... ]
  */
-__visible void __irq_entry smp_apic_timer_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_apic_timer_interrupt)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
-	/*
-	 * NOTE! We'd better ACK the irq immediately,
-	 * because timer handling can be slow.
-	 *
-	 * update_process_times() expects us to have done irq_enter().
-	 * Besides, if we don't timer interrupts ignore the global
-	 * interrupt lock, which is the WrongThing (tm) to do.
-	 */
-	entering_ack_irq();
+	ack_APIC_irq();
 	trace_local_timer_entry(LOCAL_TIMER_VECTOR);
 	local_apic_timer_interrupt();
 	trace_local_timer_exit(LOCAL_TIMER_VECTOR);
-	exiting_irq();
 
 	set_irq_regs(old_regs);
 }
@@ -2162,7 +2153,7 @@ void __init register_lapic_address(unsig
  * trigger on an entry which is routed to the common_spurious idtentry
  * point.
  *
- * Also called from smp_spurious_apic_interrupt().
+ * Also called from sysvec_spurious_apic_interrupt().
  */
 DEFINE_IDTENTRY_IRQ(spurious_interrupt)
 {
@@ -2199,17 +2190,15 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)
 	trace_spurious_apic_exit(vector);
 }
 
-__visible void smp_spurious_apic_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_spurious_apic_interrupt)
 {
-	entering_irq();
 	__spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
-	exiting_irq();
 }
 
 /*
  * This interrupt should never happen with our APIC/SMP architecture
  */
-__visible void __irq_entry smp_error_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_error_interrupt)
 {
 	static const char * const error_interrupt_reason[] = {
 		"Send CS error",		/* APIC Error Bit 0 */
@@ -2223,7 +2212,6 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)
 	};
 	u32 v, i = 0;
 
-	entering_irq();
 	trace_error_apic_entry(ERROR_APIC_VECTOR);
 
 	/* First tickle the hardware, only then report what went on. -- REW */
@@ -2247,7 +2235,6 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)
 	apic_printk(APIC_DEBUG, KERN_CONT "\n");
 
 	trace_error_apic_exit(ERROR_APIC_VECTOR);
-	exiting_irq();
 }
 
 /**
arch/x86/kernel/idt.c +4 -4
@@ -132,8 +132,8 @@ static const __initconst struct idt_data
 #endif
 
 #ifdef CONFIG_X86_LOCAL_APIC
-	INTG(LOCAL_TIMER_VECTOR,	apic_timer_interrupt),
-	INTG(X86_PLATFORM_IPI_VECTOR,	x86_platform_ipi),
+	INTG(LOCAL_TIMER_VECTOR,	asm_sysvec_apic_timer_interrupt),
+	INTG(X86_PLATFORM_IPI_VECTOR,	asm_sysvec_x86_platform_ipi),
 # ifdef CONFIG_HAVE_KVM
 	INTG(POSTED_INTR_VECTOR,	kvm_posted_intr_ipi),
 	INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi),
@@ -145,8 +145,8 @@ static const __initconst struct idt_data
 #ifdef CONFIG_X86_UV
 	INTG(UV_BAU_MESSAGE,		uv_bau_message_intr1),
 #endif
-	INTG(SPURIOUS_APIC_VECTOR,	spurious_apic_interrupt),
-	INTG(ERROR_APIC_VECTOR,		error_interrupt),
+	INTG(SPURIOUS_APIC_VECTOR,	asm_sysvec_spurious_apic_interrupt),
+	INTG(ERROR_APIC_VECTOR,		asm_sysvec_error_interrupt),
 #endif
 };
 
arch/x86/kernel/irq.c +2 -3
@@ -268,17 +268,16 @@ void (*x86_platform_ipi_callback)(void)
 /*
  * Handler for X86_PLATFORM_IPI_VECTOR.
  */
-__visible void __irq_entry smp_x86_platform_ipi(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_x86_platform_ipi)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
-	entering_ack_irq();
+	ack_APIC_irq();
 	trace_x86_platform_ipi_entry(X86_PLATFORM_IPI_VECTOR);
 	inc_irq_stat(x86_platform_ipis);
 	if (x86_platform_ipi_callback)
 		x86_platform_ipi_callback();
 	trace_x86_platform_ipi_exit(X86_PLATFORM_IPI_VECTOR);
-	exiting_irq();
 	set_irq_regs(old_regs);
 }
 #endif
#87 lore raw

[tip: x86/entry] x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     d1fa4e0e06cfcc3284f6bf51f39ae4a34f3fe434
Gitweb:        https://git.kernel.org/tip/d1fa4e0e06cfcc3284f6bf51f39ae4a34f3fe434
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:39 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC

Convert APIC interrupts to IDTENTRY_SYSVEC:

- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64-bit
- Remove the BUILD_INTERRUPT entries in 32-bit
- Remove the old prototypes

No functional change.

---
 arch/x86/entry/entry_64.S         |  6 ------
 arch/x86/include/asm/entry_arch.h |  5 -----
 arch/x86/include/asm/hw_irq.h     |  4 ----
 arch/x86/include/asm/idtentry.h   |  8 ++++++++
 arch/x86/include/asm/irq.h        |  1 -
 arch/x86/include/asm/traps.h      |  3 ---
 arch/x86/kernel/apic/apic.c       | 23 +++++------------------
 arch/x86/kernel/idt.c             |  8 ++++----
 arch/x86/kernel/irq.c             |  5 ++---
 9 files changed, 19 insertions(+), 44 deletions(-)
arch/x86/entry/entry_64.S +0 -6
index 9b7183d..25f71a0 100644
@@ -965,9 +965,6 @@ apicinterrupt3 REBOOT_VECTOR			reboot_interrupt		smp_reboot_interrupt
 apicinterrupt3 UV_BAU_MESSAGE			uv_bau_message_intr1		uv_bau_message_interrupt
 #endif
 
-apicinterrupt LOCAL_TIMER_VECTOR		apic_timer_interrupt		smp_apic_timer_interrupt
-apicinterrupt X86_PLATFORM_IPI_VECTOR		x86_platform_ipi		smp_x86_platform_ipi
-
 #ifdef CONFIG_HAVE_KVM
 apicinterrupt3 POSTED_INTR_VECTOR		kvm_posted_intr_ipi		smp_kvm_posted_intr_ipi
 apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR	kvm_posted_intr_wakeup_ipi	smp_kvm_posted_intr_wakeup_ipi
@@ -992,9 +989,6 @@ apicinterrupt CALL_FUNCTION_VECTOR		call_function_interrupt		smp_call_function_i
 apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
 
-apicinterrupt ERROR_APIC_VECTOR			error_interrupt			smp_error_interrupt
-apicinterrupt SPURIOUS_APIC_VECTOR		spurious_apic_interrupt		smp_spurious_apic_interrupt
-
 #ifdef CONFIG_IRQ_WORK
 apicinterrupt IRQ_WORK_VECTOR			irq_work_interrupt		smp_irq_work_interrupt
 #endif
arch/x86/include/asm/entry_arch.h +0 -5
index cd57ce6..d10d6d8 100644
@@ -33,11 +33,6 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)
  */
 #ifdef CONFIG_X86_LOCAL_APIC
 
-BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR)
-BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR)
-BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR)
-BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR)
-
 #ifdef CONFIG_IRQ_WORK
 BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR)
 #endif
arch/x86/include/asm/hw_irq.h +0 -4
index 3213d36..1765993 100644
@@ -29,16 +29,12 @@
 #include <asm/sections.h>
 
 /* Interrupt handlers registered during init_IRQ */
-extern asmlinkage void apic_timer_interrupt(void);
-extern asmlinkage void x86_platform_ipi(void);
 extern asmlinkage void kvm_posted_intr_ipi(void);
 extern asmlinkage void kvm_posted_intr_wakeup_ipi(void);
 extern asmlinkage void kvm_posted_intr_nested_ipi(void);
-extern asmlinkage void error_interrupt(void);
 extern asmlinkage void irq_work_interrupt(void);
 extern asmlinkage void uv_bau_message_intr1(void);
 
-extern asmlinkage void spurious_apic_interrupt(void);
 extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
 
arch/x86/include/asm/idtentry.h +8 -0
index 31948e6..3e58480 100644
@@ -569,6 +569,14 @@ DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER,	common_interrupt);
 DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER,	spurious_interrupt);
 #endif
 
+/* System vector entry points */
+#ifdef CONFIG_X86_LOCAL_APIC
+DECLARE_IDTENTRY_SYSVEC(ERROR_APIC_VECTOR,		sysvec_error_interrupt);
+DECLARE_IDTENTRY_SYSVEC(SPURIOUS_APIC_VECTOR,		sysvec_spurious_apic_interrupt);
+DECLARE_IDTENTRY_SYSVEC(LOCAL_TIMER_VECTOR,		sysvec_apic_timer_interrupt);
+DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR,	sysvec_x86_platform_ipi);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/irq.h +0 -1
index 67aa1e2..c7c43e8 100644
@@ -46,7 +46,6 @@ extern void __init init_IRQ(void);
 void arch_trigger_cpumask_backtrace(const struct cpumask *mask,
 				    bool exclude_self);
 
-extern __visible void smp_x86_platform_ipi(struct pt_regs *regs);
 #define arch_trigger_cpumask_backtrace arch_trigger_cpumask_backtrace
 #endif
 
arch/x86/include/asm/traps.h +0 -3
index 97e6945..933934c 100644
@@ -40,9 +40,6 @@ asmlinkage void smp_threshold_interrupt(struct pt_regs *regs);
 asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);
 #endif
 
-void smp_apic_timer_interrupt(struct pt_regs *regs);
-void smp_error_interrupt(struct pt_regs *regs);
-void smp_spurious_apic_interrupt(struct pt_regs *regs);
 asmlinkage void smp_irq_move_cleanup_interrupt(void);
 
 #ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +5 -18
index 66c3cfc..6a8e9f3 100644
@@ -1121,23 +1121,14 @@ static void local_apic_timer_interrupt(void)
  * [ if a single-CPU system runs an SMP kernel then we call the local
  *   interrupt as well. Thus we cannot inline the local irq ... ]
  */
-__visible void __irq_entry smp_apic_timer_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_apic_timer_interrupt)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
-	/*
-	 * NOTE! We'd better ACK the irq immediately,
-	 * because timer handling can be slow.
-	 *
-	 * update_process_times() expects us to have done irq_enter().
-	 * Besides, if we don't timer interrupts ignore the global
-	 * interrupt lock, which is the WrongThing (tm) to do.
-	 */
-	entering_ack_irq();
+	ack_APIC_irq();
 	trace_local_timer_entry(LOCAL_TIMER_VECTOR);
 	local_apic_timer_interrupt();
 	trace_local_timer_exit(LOCAL_TIMER_VECTOR);
-	exiting_irq();
 
 	set_irq_regs(old_regs);
 }
@@ -2162,7 +2153,7 @@ void __init register_lapic_address(unsigned long address)
  * trigger on an entry which is routed to the common_spurious idtentry
  * point.
  *
- * Also called from smp_spurious_apic_interrupt().
+ * Also called from sysvec_spurious_apic_interrupt().
  */
 DEFINE_IDTENTRY_IRQ(spurious_interrupt)
 {
@@ -2199,17 +2190,15 @@ out:
 	trace_spurious_apic_exit(vector);
 }
 
-__visible void smp_spurious_apic_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_spurious_apic_interrupt)
 {
-	entering_irq();
 	__spurious_interrupt(regs, SPURIOUS_APIC_VECTOR);
-	exiting_irq();
 }
 
 /*
  * This interrupt should never happen with our APIC/SMP architecture
  */
-__visible void __irq_entry smp_error_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_error_interrupt)
 {
 	static const char * const error_interrupt_reason[] = {
 		"Send CS error",		/* APIC Error Bit 0 */
@@ -2223,7 +2212,6 @@ __visible void __irq_entry smp_error_interrupt(struct pt_regs *regs)
 	};
 	u32 v, i = 0;
 
-	entering_irq();
 	trace_error_apic_entry(ERROR_APIC_VECTOR);
 
 	/* First tickle the hardware, only then report what went on. -- REW */
@@ -2247,7 +2235,6 @@ __visible void __irq_entry smp_error_interrupt(struct pt_regs *regs)
 	apic_printk(APIC_DEBUG, KERN_CONT "\n");
 
 	trace_error_apic_exit(ERROR_APIC_VECTOR);
-	exiting_irq();
 }
 
 /**
arch/x86/kernel/idt.c +4 -4
index 20408e3..93c1b27 100644
@@ -132,8 +132,8 @@ static const __initconst struct idt_data apic_idts[] = {
 #endif
 
 #ifdef CONFIG_X86_LOCAL_APIC
-	INTG(LOCAL_TIMER_VECTOR,	apic_timer_interrupt),
-	INTG(X86_PLATFORM_IPI_VECTOR,	x86_platform_ipi),
+	INTG(LOCAL_TIMER_VECTOR,	asm_sysvec_apic_timer_interrupt),
+	INTG(X86_PLATFORM_IPI_VECTOR,	asm_sysvec_x86_platform_ipi),
 # ifdef CONFIG_HAVE_KVM
 	INTG(POSTED_INTR_VECTOR,	kvm_posted_intr_ipi),
 	INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi),
@@ -145,8 +145,8 @@ static const __initconst struct idt_data apic_idts[] = {
 #ifdef CONFIG_X86_UV
 	INTG(UV_BAU_MESSAGE,		uv_bau_message_intr1),
 #endif
-	INTG(SPURIOUS_APIC_VECTOR,	spurious_apic_interrupt),
-	INTG(ERROR_APIC_VECTOR,		error_interrupt),
+	INTG(SPURIOUS_APIC_VECTOR,	asm_sysvec_spurious_apic_interrupt),
+	INTG(ERROR_APIC_VECTOR,		asm_sysvec_error_interrupt),
 #endif
 };
 
arch/x86/kernel/irq.c +2 -3
index c449b84..7e30052 100644
@@ -268,17 +268,16 @@ void (*x86_platform_ipi_callback)(void) = NULL;
 /*
  * Handler for X86_PLATFORM_IPI_VECTOR.
  */
-__visible void __irq_entry smp_x86_platform_ipi(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_x86_platform_ipi)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
-	entering_ack_irq();
+	ack_APIC_irq();
 	trace_x86_platform_ipi_entry(X86_PLATFORM_IPI_VECTOR);
 	inc_irq_stat(x86_platform_ipis);
 	if (x86_platform_ipi_callback)
 		x86_platform_ipi_callback();
 	trace_x86_platform_ipi_exit(X86_PLATFORM_IPI_VECTOR);
-	exiting_irq();
 	set_irq_regs(old_regs);
 }
 #endif
#88 lore raw

PATCH v9 27/39 x86/entry: Convert SMP system vectors to IDTENTRY_SYSVEC

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Convert SMP system vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes

No functional change.

---
 arch/x86/entry/entry_64.S         |    7 -------
 arch/x86/include/asm/entry_arch.h |    4 ----
 arch/x86/include/asm/hw_irq.h     |    5 -----
 arch/x86/include/asm/idtentry.h   |    7 +++++++
 arch/x86/include/asm/traps.h      |    2 --
 arch/x86/kernel/apic/vector.c     |    5 ++---
 arch/x86/kernel/idt.c             |   10 +++++-----
 arch/x86/kernel/smp.c             |   18 +++++++-----------
 8 files changed, 21 insertions(+), 37 deletions(-)
arch/x86/entry/entry_64.S +0 -7
@@ -956,11 +956,6 @@ apicinterrupt3 \num \sym \do_sym
 POP_SECTION_IRQENTRY
 .endm
 
-#ifdef CONFIG_SMP
-apicinterrupt3 IRQ_MOVE_CLEANUP_VECTOR		irq_move_cleanup_interrupt	smp_irq_move_cleanup_interrupt
-apicinterrupt3 REBOOT_VECTOR			reboot_interrupt		smp_reboot_interrupt
-#endif
-
 #ifdef CONFIG_X86_UV
 apicinterrupt3 UV_BAU_MESSAGE			uv_bau_message_intr1		uv_bau_message_interrupt
 #endif
@@ -984,8 +979,6 @@ apicinterrupt THERMAL_APIC_VECTOR		therm
 #endif
 
 #ifdef CONFIG_SMP
-apicinterrupt CALL_FUNCTION_SINGLE_VECTOR	call_function_single_interrupt	smp_call_function_single_interrupt
-apicinterrupt CALL_FUNCTION_VECTOR		call_function_interrupt		smp_call_function_interrupt
 apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
 
arch/x86/include/asm/entry_arch.h +0 -4
@@ -12,10 +12,6 @@
  */
 #ifdef CONFIG_SMP
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
-BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)
-BUILD_INTERRUPT(call_function_single_interrupt,CALL_FUNCTION_SINGLE_VECTOR)
-BUILD_INTERRUPT(irq_move_cleanup_interrupt, IRQ_MOVE_CLEANUP_VECTOR)
-BUILD_INTERRUPT(reboot_interrupt, REBOOT_VECTOR)
 #endif
 
 #ifdef CONFIG_HAVE_KVM
arch/x86/include/asm/hw_irq.h +0 -5
@@ -38,14 +38,9 @@ extern asmlinkage void uv_bau_message_in
 extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
 
-extern asmlinkage void irq_move_cleanup_interrupt(void);
-extern asmlinkage void reboot_interrupt(void);
 extern asmlinkage void threshold_interrupt(void);
 extern asmlinkage void deferred_error_interrupt(void);
 
-extern asmlinkage void call_function_interrupt(void);
-extern asmlinkage void call_function_single_interrupt(void);
-
 #ifdef	CONFIG_X86_LOCAL_APIC
 struct irq_data;
 struct pci_dev;
arch/x86/include/asm/idtentry.h +7 -0
@@ -573,6 +573,13 @@ DECLARE_IDTENTRY_SYSVEC(LOCAL_TIMER_VECT
 DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR,	sysvec_x86_platform_ipi);
 #endif
 
+#ifdef CONFIG_SMP
+DECLARE_IDTENTRY_SYSVEC(IRQ_MOVE_CLEANUP_VECTOR,	sysvec_irq_move_cleanup);
+DECLARE_IDTENTRY_SYSVEC(REBOOT_VECTOR,			sysvec_reboot);
+DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,	sysvec_call_function_single);
+DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_VECTOR,		sysvec_call_function);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/traps.h +0 -2
@@ -40,8 +40,6 @@ asmlinkage void smp_threshold_interrupt(
 asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);
 #endif
 
-asmlinkage void smp_irq_move_cleanup_interrupt(void);
-
 #ifdef CONFIG_VMAP_STACK
 void __noreturn handle_stack_overflow(const char *message,
 				      struct pt_regs *regs,
arch/x86/kernel/apic/vector.c +2 -3
@@ -861,13 +861,13 @@ static void free_moved_vector(struct api
 	apicd->move_in_progress = 0;
 }
 
-asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void)
+DEFINE_IDTENTRY_SYSVEC(sysvec_irq_move_cleanup)
 {
 	struct hlist_head *clhead = this_cpu_ptr(&cleanup_list);
 	struct apic_chip_data *apicd;
 	struct hlist_node *tmp;
 
-	entering_ack_irq();
+	ack_APIC_irq();
 	/* Prevent vectors vanishing under us */
 	raw_spin_lock(&vector_lock);
 
@@ -892,7 +892,6 @@ asmlinkage __visible void __irq_entry sm
 	}
 
 	raw_spin_unlock(&vector_lock);
-	exiting_irq();
 }
 
 static void __send_cleanup_vector(struct apic_chip_data *apicd)
arch/x86/kernel/idt.c +5 -5
@@ -112,11 +112,11 @@ static const __initconst struct idt_data
  */
 static const __initconst struct idt_data apic_idts[] = {
 #ifdef CONFIG_SMP
-	INTG(RESCHEDULE_VECTOR,		reschedule_interrupt),
-	INTG(CALL_FUNCTION_VECTOR,	call_function_interrupt),
-	INTG(CALL_FUNCTION_SINGLE_VECTOR, call_function_single_interrupt),
-	INTG(IRQ_MOVE_CLEANUP_VECTOR,	irq_move_cleanup_interrupt),
-	INTG(REBOOT_VECTOR,		reboot_interrupt),
+	INTG(RESCHEDULE_VECTOR,			reschedule_interrupt),
+	INTG(CALL_FUNCTION_VECTOR,		asm_sysvec_call_function),
+	INTG(CALL_FUNCTION_SINGLE_VECTOR,	asm_sysvec_call_function_single),
+	INTG(IRQ_MOVE_CLEANUP_VECTOR,		asm_sysvec_irq_move_cleanup),
+	INTG(REBOOT_VECTOR,			asm_sysvec_reboot),
 #endif
 
 #ifdef CONFIG_X86_THERMAL_VECTOR
arch/x86/kernel/smp.c +7 -11
@@ -27,6 +27,7 @@
 #include <asm/mmu_context.h>
 #include <asm/proto.h>
 #include <asm/apic.h>
+#include <asm/idtentry.h>
 #include <asm/nmi.h>
 #include <asm/mce.h>
 #include <asm/trace/irq_vectors.h>
@@ -130,13 +131,11 @@ static int smp_stop_nmi_callback(unsigne
 /*
  * this function calls the 'stop' function on all other CPUs in the system.
  */
-
-asmlinkage __visible void smp_reboot_interrupt(void)
+DEFINE_IDTENTRY_SYSVEC(sysvec_reboot)
 {
-	ipi_entering_ack_irq();
+	ack_APIC_irq();
 	cpu_emergency_vmxoff();
 	stop_this_cpu(NULL);
-	irq_exit();
 }
 
 static int register_stop_handler(void)
@@ -227,7 +226,6 @@ static void native_stop_other_cpus(int w
 {
 	ack_APIC_irq();
 	inc_irq_stat(irq_resched_count);
-	kvm_set_cpu_l1tf_flush_l1d();
 
 	if (trace_resched_ipi_enabled()) {
 		/*
@@ -244,24 +242,22 @@ static void native_stop_other_cpus(int w
 	scheduler_ipi();
 }
 
-__visible void __irq_entry smp_call_function_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_call_function)
 {
-	ipi_entering_ack_irq();
+	ack_APIC_irq();
 	trace_call_function_entry(CALL_FUNCTION_VECTOR);
 	inc_irq_stat(irq_call_count);
 	generic_smp_call_function_interrupt();
 	trace_call_function_exit(CALL_FUNCTION_VECTOR);
-	exiting_irq();
 }
 
-__visible void __irq_entry smp_call_function_single_interrupt(struct pt_regs *r)
+DEFINE_IDTENTRY_SYSVEC(sysvec_call_function_single)
 {
-	ipi_entering_ack_irq();
+	ack_APIC_irq();
 	trace_call_function_single_entry(CALL_FUNCTION_SINGLE_VECTOR);
 	inc_irq_stat(irq_call_count);
 	generic_smp_call_function_single_interrupt();
 	trace_call_function_single_exit(CALL_FUNCTION_SINGLE_VECTOR);
-	exiting_irq();
 }
 
 static int __init nonmi_ipi_setup(char *str)
#89 lore raw

[tip: x86/entry] x86/entry: Convert SMP system vectors to IDTENTRY_SYSVEC

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     8f2d02673efc50ad7075fbbfce465766cf8f1a20
Gitweb:        https://git.kernel.org/tip/8f2d02673efc50ad7075fbbfce465766cf8f1a20
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:40 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Convert SMP system vectors to IDTENTRY_SYSVEC

Convert SMP system vectors to IDTENTRY_SYSVEC:

- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64-bit
- Remove the BUILD_INTERRUPT entries in 32-bit
- Remove the old prototypes

No functional change.

---
 arch/x86/entry/entry_64.S         |  7 -------
 arch/x86/include/asm/entry_arch.h |  4 ----
 arch/x86/include/asm/hw_irq.h     |  5 -----
 arch/x86/include/asm/idtentry.h   |  7 +++++++
 arch/x86/include/asm/traps.h      |  2 --
 arch/x86/kernel/apic/vector.c     |  5 ++---
 arch/x86/kernel/idt.c             | 10 +++++-----
 arch/x86/kernel/smp.c             | 18 +++++++-----------
 8 files changed, 21 insertions(+), 37 deletions(-)
arch/x86/entry/entry_64.S +0 -7
index 25f71a0..f3ccb27 100644
@@ -956,11 +956,6 @@ apicinterrupt3 \num \sym \do_sym
 POP_SECTION_IRQENTRY
 .endm
 
-#ifdef CONFIG_SMP
-apicinterrupt3 IRQ_MOVE_CLEANUP_VECTOR		irq_move_cleanup_interrupt	smp_irq_move_cleanup_interrupt
-apicinterrupt3 REBOOT_VECTOR			reboot_interrupt		smp_reboot_interrupt
-#endif
-
 #ifdef CONFIG_X86_UV
 apicinterrupt3 UV_BAU_MESSAGE			uv_bau_message_intr1		uv_bau_message_interrupt
 #endif
@@ -984,8 +979,6 @@ apicinterrupt THERMAL_APIC_VECTOR		thermal_interrupt		smp_thermal_interrupt
 #endif
 
 #ifdef CONFIG_SMP
-apicinterrupt CALL_FUNCTION_SINGLE_VECTOR	call_function_single_interrupt	smp_call_function_single_interrupt
-apicinterrupt CALL_FUNCTION_VECTOR		call_function_interrupt		smp_call_function_interrupt
 apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
 
arch/x86/include/asm/entry_arch.h +0 -4
index d10d6d8..2e2055b 100644
@@ -12,10 +12,6 @@
  */
 #ifdef CONFIG_SMP
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
-BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR)
-BUILD_INTERRUPT(call_function_single_interrupt,CALL_FUNCTION_SINGLE_VECTOR)
-BUILD_INTERRUPT(irq_move_cleanup_interrupt, IRQ_MOVE_CLEANUP_VECTOR)
-BUILD_INTERRUPT(reboot_interrupt, REBOOT_VECTOR)
 #endif
 
 #ifdef CONFIG_HAVE_KVM
arch/x86/include/asm/hw_irq.h +0 -5
index 1765993..36a3869 100644
@@ -38,14 +38,9 @@ extern asmlinkage void uv_bau_message_intr1(void);
 extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
 
-extern asmlinkage void irq_move_cleanup_interrupt(void);
-extern asmlinkage void reboot_interrupt(void);
 extern asmlinkage void threshold_interrupt(void);
 extern asmlinkage void deferred_error_interrupt(void);
 
-extern asmlinkage void call_function_interrupt(void);
-extern asmlinkage void call_function_single_interrupt(void);
-
 #ifdef	CONFIG_X86_LOCAL_APIC
 struct irq_data;
 struct pci_dev;
arch/x86/include/asm/idtentry.h +7 -0
index 3e58480..82cca1f 100644
@@ -577,6 +577,13 @@ DECLARE_IDTENTRY_SYSVEC(LOCAL_TIMER_VECTOR,		sysvec_apic_timer_interrupt);
 DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR,	sysvec_x86_platform_ipi);
 #endif
 
+#ifdef CONFIG_SMP
+DECLARE_IDTENTRY_SYSVEC(IRQ_MOVE_CLEANUP_VECTOR,	sysvec_irq_move_cleanup);
+DECLARE_IDTENTRY_SYSVEC(REBOOT_VECTOR,			sysvec_reboot);
+DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,	sysvec_call_function_single);
+DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_VECTOR,		sysvec_call_function);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/traps.h +0 -2
index 933934c..0c40f37 100644
@@ -40,8 +40,6 @@ asmlinkage void smp_threshold_interrupt(struct pt_regs *regs);
 asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);
 #endif
 
-asmlinkage void smp_irq_move_cleanup_interrupt(void);
-
 #ifdef CONFIG_VMAP_STACK
 void __noreturn handle_stack_overflow(const char *message,
 				      struct pt_regs *regs,
arch/x86/kernel/apic/vector.c +2 -3
index 67768e5..c48be6e 100644
@@ -861,13 +861,13 @@ static void free_moved_vector(struct apic_chip_data *apicd)
 	apicd->move_in_progress = 0;
 }
 
-asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void)
+DEFINE_IDTENTRY_SYSVEC(sysvec_irq_move_cleanup)
 {
 	struct hlist_head *clhead = this_cpu_ptr(&cleanup_list);
 	struct apic_chip_data *apicd;
 	struct hlist_node *tmp;
 
-	entering_ack_irq();
+	ack_APIC_irq();
 	/* Prevent vectors vanishing under us */
 	raw_spin_lock(&vector_lock);
 
@@ -892,7 +892,6 @@ asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void)
 	}
 
 	raw_spin_unlock(&vector_lock);
-	exiting_irq();
 }
 
 static void __send_cleanup_vector(struct apic_chip_data *apicd)
arch/x86/kernel/idt.c +5 -5
index 93c1b27..018a542 100644
@@ -112,11 +112,11 @@ static const __initconst struct idt_data def_idts[] = {
  */
 static const __initconst struct idt_data apic_idts[] = {
 #ifdef CONFIG_SMP
-	INTG(RESCHEDULE_VECTOR,		reschedule_interrupt),
-	INTG(CALL_FUNCTION_VECTOR,	call_function_interrupt),
-	INTG(CALL_FUNCTION_SINGLE_VECTOR, call_function_single_interrupt),
-	INTG(IRQ_MOVE_CLEANUP_VECTOR,	irq_move_cleanup_interrupt),
-	INTG(REBOOT_VECTOR,		reboot_interrupt),
+	INTG(RESCHEDULE_VECTOR,			reschedule_interrupt),
+	INTG(CALL_FUNCTION_VECTOR,		asm_sysvec_call_function),
+	INTG(CALL_FUNCTION_SINGLE_VECTOR,	asm_sysvec_call_function_single),
+	INTG(IRQ_MOVE_CLEANUP_VECTOR,		asm_sysvec_irq_move_cleanup),
+	INTG(REBOOT_VECTOR,			asm_sysvec_reboot),
 #endif
 
 #ifdef CONFIG_X86_THERMAL_VECTOR
arch/x86/kernel/smp.c +7 -11
index b8d4e9c..e5647da 100644
@@ -27,6 +27,7 @@
 #include <asm/mmu_context.h>
 #include <asm/proto.h>
 #include <asm/apic.h>
+#include <asm/idtentry.h>
 #include <asm/nmi.h>
 #include <asm/mce.h>
 #include <asm/trace/irq_vectors.h>
@@ -130,13 +131,11 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs)
 /*
  * this function calls the 'stop' function on all other CPUs in the system.
  */
-
-asmlinkage __visible void smp_reboot_interrupt(void)
+DEFINE_IDTENTRY_SYSVEC(sysvec_reboot)
 {
-	ipi_entering_ack_irq();
+	ack_APIC_irq();
 	cpu_emergency_vmxoff();
 	stop_this_cpu(NULL);
-	irq_exit();
 }
 
 static int register_stop_handler(void)
@@ -227,7 +226,6 @@ __visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs)
 {
 	ack_APIC_irq();
 	inc_irq_stat(irq_resched_count);
-	kvm_set_cpu_l1tf_flush_l1d();
 
 	if (trace_resched_ipi_enabled()) {
 		/*
@@ -244,24 +242,22 @@ __visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs)
 	scheduler_ipi();
 }
 
-__visible void __irq_entry smp_call_function_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_call_function)
 {
-	ipi_entering_ack_irq();
+	ack_APIC_irq();
 	trace_call_function_entry(CALL_FUNCTION_VECTOR);
 	inc_irq_stat(irq_call_count);
 	generic_smp_call_function_interrupt();
 	trace_call_function_exit(CALL_FUNCTION_VECTOR);
-	exiting_irq();
 }
 
-__visible void __irq_entry smp_call_function_single_interrupt(struct pt_regs *r)
+DEFINE_IDTENTRY_SYSVEC(sysvec_call_function_single)
 {
-	ipi_entering_ack_irq();
+	ack_APIC_irq();
 	trace_call_function_single_entry(CALL_FUNCTION_SINGLE_VECTOR);
 	inc_irq_stat(irq_call_count);
 	generic_smp_call_function_single_interrupt();
 	trace_call_function_single_exit(CALL_FUNCTION_SINGLE_VECTOR);
-	exiting_irq();
 }
 
 static int __init nonmi_ipi_setup(char *str)
#90 lore raw

PATCH v9 28/39 x86/entry: Convert various system vectors

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Convert various system vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes

No functional change.

---
 arch/x86/entry/entry_64.S             |   19 -------------------
 arch/x86/include/asm/apic.h           |   13 -------------
 arch/x86/include/asm/entry_arch.h     |   25 -------------------------
 arch/x86/include/asm/hw_irq.h         |    6 ------
 arch/x86/include/asm/idtentry.h       |   22 ++++++++++++++++++++++
 arch/x86/include/asm/irq_work.h       |    1 -
 arch/x86/include/asm/traps.h          |    5 -----
 arch/x86/include/asm/uv/uv_bau.h      |    8 ++------
 arch/x86/kernel/cpu/mce/amd.c         |    5 ++---
 arch/x86/kernel/cpu/mce/therm_throt.c |    5 ++---
 arch/x86/kernel/cpu/mce/threshold.c   |    5 ++---
 arch/x86/kernel/idt.c                 |   28 ++++++++++++++--------------
 arch/x86/kernel/irq_work.c            |    6 +++---
 arch/x86/platform/uv/tlb_uv.c         |    2 +-
 14 files changed, 48 insertions(+), 102 deletions(-)
arch/x86/entry/entry_64.S +0 -19
@@ -956,9 +956,6 @@ apicinterrupt3 \num \sym \do_sym
 POP_SECTION_IRQENTRY
 .endm
 
-#ifdef CONFIG_X86_UV
-apicinterrupt3 UV_BAU_MESSAGE			uv_bau_message_intr1		uv_bau_message_interrupt
-#endif
 
 #ifdef CONFIG_HAVE_KVM
 apicinterrupt3 POSTED_INTR_VECTOR		kvm_posted_intr_ipi		smp_kvm_posted_intr_ipi
@@ -966,26 +963,10 @@ apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR
 apicinterrupt3 POSTED_INTR_NESTED_VECTOR	kvm_posted_intr_nested_ipi	smp_kvm_posted_intr_nested_ipi
 #endif
 
-#ifdef CONFIG_X86_MCE_THRESHOLD
-apicinterrupt THRESHOLD_APIC_VECTOR		threshold_interrupt		smp_threshold_interrupt
-#endif
-
-#ifdef CONFIG_X86_MCE_AMD
-apicinterrupt DEFERRED_ERROR_VECTOR		deferred_error_interrupt	smp_deferred_error_interrupt
-#endif
-
-#ifdef CONFIG_X86_THERMAL_VECTOR
-apicinterrupt THERMAL_APIC_VECTOR		thermal_interrupt		smp_thermal_interrupt
-#endif
-
 #ifdef CONFIG_SMP
 apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
 
-#ifdef CONFIG_IRQ_WORK
-apicinterrupt IRQ_WORK_VECTOR			irq_work_interrupt		smp_irq_work_interrupt
-#endif
-
 /*
  * Reload gs selector with exception handling
  * edi:  new selector
arch/x86/include/asm/apic.h +0 -13
@@ -534,24 +534,11 @@ static inline void entering_ack_irq(void
 	ack_APIC_irq();
 }
 
-static inline void ipi_entering_ack_irq(void)
-{
-	irq_enter();
-	ack_APIC_irq();
-	kvm_set_cpu_l1tf_flush_l1d();
-}
-
 static inline void exiting_irq(void)
 {
 	irq_exit();
 }
 
-static inline void exiting_ack_irq(void)
-{
-	ack_APIC_irq();
-	irq_exit();
-}
-
 extern void ioapic_zap_locks(void);
 
 #endif /* _ASM_X86_APIC_H */
arch/x86/include/asm/entry_arch.h +0 -25
@@ -20,28 +20,3 @@ BUILD_INTERRUPT(kvm_posted_intr_wakeup_i
 BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)
 #endif
 
-/*
- * every pentium local APIC has two 'local interrupts', with a
- * soft-definable vector attached to both interrupts, one of
- * which is a timer interrupt, the other one is error counter
- * overflow. Linux uses the local APIC timer interrupt to get
- * a much simpler SMP time architecture:
- */
-#ifdef CONFIG_X86_LOCAL_APIC
-
-#ifdef CONFIG_IRQ_WORK
-BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR)
-#endif
-
-#ifdef CONFIG_X86_THERMAL_VECTOR
-BUILD_INTERRUPT(thermal_interrupt,THERMAL_APIC_VECTOR)
-#endif
-
-#ifdef CONFIG_X86_MCE_THRESHOLD
-BUILD_INTERRUPT(threshold_interrupt,THRESHOLD_APIC_VECTOR)
-#endif
-
-#ifdef CONFIG_X86_MCE_AMD
-BUILD_INTERRUPT(deferred_error_interrupt, DEFERRED_ERROR_VECTOR)
-#endif
-#endif
arch/x86/include/asm/hw_irq.h +0 -6
@@ -32,15 +32,9 @@
 extern asmlinkage void kvm_posted_intr_ipi(void);
 extern asmlinkage void kvm_posted_intr_wakeup_ipi(void);
 extern asmlinkage void kvm_posted_intr_nested_ipi(void);
-extern asmlinkage void irq_work_interrupt(void);
-extern asmlinkage void uv_bau_message_intr1(void);
 
-extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
 
-extern asmlinkage void threshold_interrupt(void);
-extern asmlinkage void deferred_error_interrupt(void);
-
 #ifdef	CONFIG_X86_LOCAL_APIC
 struct irq_data;
 struct pci_dev;
arch/x86/include/asm/idtentry.h +22 -0
@@ -580,6 +580,28 @@ DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SI
 DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_VECTOR,		sysvec_call_function);
 #endif
 
+#ifdef CONFIG_X86_LOCAL_APIC
+# ifdef CONFIG_X86_UV
+DECLARE_IDTENTRY_SYSVEC(UV_BAU_MESSAGE,			sysvec_uv_bau_message);
+# endif
+
+# ifdef CONFIG_X86_MCE_THRESHOLD
+DECLARE_IDTENTRY_SYSVEC(THRESHOLD_APIC_VECTOR,		sysvec_threshold);
+# endif
+
+# ifdef CONFIG_X86_MCE_AMD
+DECLARE_IDTENTRY_SYSVEC(DEFERRED_ERROR_VECTOR,		sysvec_deferred_error);
+# endif
+
+# ifdef CONFIG_X86_THERMAL_VECTOR
+DECLARE_IDTENTRY_SYSVEC(THERMAL_APIC_VECTOR,		sysvec_thermal);
+# endif
+
+# ifdef CONFIG_IRQ_WORK
+DECLARE_IDTENTRY_SYSVEC(IRQ_WORK_VECTOR,		sysvec_irq_work);
+# endif
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/irq_work.h +0 -1
@@ -10,7 +10,6 @@ static inline bool arch_irq_work_has_int
 	return boot_cpu_has(X86_FEATURE_APIC);
 }
 extern void arch_irq_work_raise(void);
-extern __visible void smp_irq_work_interrupt(struct pt_regs *regs);
 #else
 static inline bool arch_irq_work_has_interrupt(void)
 {
arch/x86/include/asm/traps.h +0 -5
@@ -34,11 +34,6 @@ static inline int get_si_code(unsigned l
 extern int panic_on_unrecovered_nmi;
 
 void math_emulate(struct math_emu_info *);
-#ifndef CONFIG_X86_32
-asmlinkage void smp_thermal_interrupt(struct pt_regs *regs);
-asmlinkage void smp_threshold_interrupt(struct pt_regs *regs);
-asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);
-#endif
 
 #ifdef CONFIG_VMAP_STACK
 void __noreturn handle_stack_overflow(const char *message,
arch/x86/include/asm/uv/uv_bau.h +2 -6
@@ -12,6 +12,8 @@
 #define _ASM_X86_UV_UV_BAU_H
 
 #include <linux/bitmap.h>
+#include <asm/idtentry.h>
+
 #define BITSPERBYTE 8
 
 /*
@@ -799,12 +801,6 @@ static inline void bau_cpubits_clear(str
 	bitmap_zero(&dstp->bits, nbits);
 }
 
-extern void uv_bau_message_intr1(void);
-#ifdef CONFIG_TRACING
-#define trace_uv_bau_message_intr1 uv_bau_message_intr1
-#endif
-extern void uv_bau_timeout_intr1(void);
-
 struct atomic_short {
 	short counter;
 };
arch/x86/kernel/cpu/mce/amd.c +2 -3
@@ -907,14 +907,13 @@ static void __log_error(unsigned int ban
 	mce_log(&m);
 }
 
-asmlinkage __visible void __irq_entry smp_deferred_error_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_deferred_error)
 {
-	entering_irq();
 	trace_deferred_error_apic_entry(DEFERRED_ERROR_VECTOR);
 	inc_irq_stat(irq_deferred_error_count);
 	deferred_error_int_vector();
 	trace_deferred_error_apic_exit(DEFERRED_ERROR_VECTOR);
-	exiting_ack_irq();
+	ack_APIC_irq();
 }
 
 /*
arch/x86/kernel/cpu/mce/therm_throt.c +2 -3
@@ -614,14 +614,13 @@ static void unexpected_thermal_interrupt
 
 static void (*smp_thermal_vector)(void) = unexpected_thermal_interrupt;
 
-asmlinkage __visible void __irq_entry smp_thermal_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_thermal)
 {
-	entering_irq();
 	trace_thermal_apic_entry(THERMAL_APIC_VECTOR);
 	inc_irq_stat(irq_thermal_count);
 	smp_thermal_vector();
 	trace_thermal_apic_exit(THERMAL_APIC_VECTOR);
-	exiting_ack_irq();
+	ack_APIC_irq();
 }
 
 /* Thermal monitoring depends on APIC, ACPI and clock modulation */
arch/x86/kernel/cpu/mce/threshold.c +2 -3
@@ -21,12 +21,11 @@ static void default_threshold_interrupt(
 
 void (*mce_threshold_vector)(void) = default_threshold_interrupt;
 
-asmlinkage __visible void __irq_entry smp_threshold_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_threshold)
 {
-	entering_irq();
 	trace_threshold_apic_entry(THRESHOLD_APIC_VECTOR);
 	inc_irq_stat(irq_threshold_count);
 	mce_threshold_vector();
 	trace_threshold_apic_exit(THRESHOLD_APIC_VECTOR);
-	exiting_ack_irq();
+	ack_APIC_irq();
 }
arch/x86/kernel/idt.c +14 -14
@@ -120,33 +120,33 @@ static const __initconst struct idt_data
 #endif
 
 #ifdef CONFIG_X86_THERMAL_VECTOR
-	INTG(THERMAL_APIC_VECTOR,	thermal_interrupt),
+	INTG(THERMAL_APIC_VECTOR,		asm_sysvec_thermal),
 #endif
 
 #ifdef CONFIG_X86_MCE_THRESHOLD
-	INTG(THRESHOLD_APIC_VECTOR,	threshold_interrupt),
+	INTG(THRESHOLD_APIC_VECTOR,		asm_sysvec_threshold),
 #endif
 
 #ifdef CONFIG_X86_MCE_AMD
-	INTG(DEFERRED_ERROR_VECTOR,	deferred_error_interrupt),
+	INTG(DEFERRED_ERROR_VECTOR,		asm_sysvec_deferred_error),
 #endif
 
 #ifdef CONFIG_X86_LOCAL_APIC
-	INTG(LOCAL_TIMER_VECTOR,	asm_sysvec_apic_timer_interrupt),
-	INTG(X86_PLATFORM_IPI_VECTOR,	asm_sysvec_x86_platform_ipi),
+	INTG(LOCAL_TIMER_VECTOR,		asm_sysvec_apic_timer_interrupt),
+	INTG(X86_PLATFORM_IPI_VECTOR,		asm_sysvec_x86_platform_ipi),
 # ifdef CONFIG_HAVE_KVM
-	INTG(POSTED_INTR_VECTOR,	kvm_posted_intr_ipi),
-	INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi),
-	INTG(POSTED_INTR_NESTED_VECTOR, kvm_posted_intr_nested_ipi),
+	INTG(POSTED_INTR_VECTOR,		kvm_posted_intr_ipi),
+	INTG(POSTED_INTR_WAKEUP_VECTOR,		kvm_posted_intr_wakeup_ipi),
+	INTG(POSTED_INTR_NESTED_VECTOR,		kvm_posted_intr_nested_ipi),
 # endif
 # ifdef CONFIG_IRQ_WORK
-	INTG(IRQ_WORK_VECTOR,		irq_work_interrupt),
+	INTG(IRQ_WORK_VECTOR,			asm_sysvec_irq_work),
 # endif
-#ifdef CONFIG_X86_UV
-	INTG(UV_BAU_MESSAGE,		uv_bau_message_intr1),
-#endif
-	INTG(SPURIOUS_APIC_VECTOR,	asm_sysvec_spurious_apic_interrupt),
-	INTG(ERROR_APIC_VECTOR,		asm_sysvec_error_interrupt),
+# ifdef CONFIG_X86_UV
+	INTG(UV_BAU_MESSAGE,			asm_sysvec_uv_bau_message),
+# endif
+	INTG(SPURIOUS_APIC_VECTOR,		asm_sysvec_spurious_apic_interrupt),
+	INTG(ERROR_APIC_VECTOR,			asm_sysvec_error_interrupt),
 #endif
 };
 
arch/x86/kernel/irq_work.c +3 -3
@@ -9,18 +9,18 @@
 #include <linux/irq_work.h>
 #include <linux/hardirq.h>
 #include <asm/apic.h>
+#include <asm/idtentry.h>
 #include <asm/trace/irq_vectors.h>
 #include <linux/interrupt.h>
 
 #ifdef CONFIG_X86_LOCAL_APIC
-__visible void __irq_entry smp_irq_work_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_irq_work)
 {
-	ipi_entering_ack_irq();
+	ack_APIC_irq();
 	trace_irq_work_entry(IRQ_WORK_VECTOR);
 	inc_irq_stat(apic_irq_work_irqs);
 	irq_work_run();
 	trace_irq_work_exit(IRQ_WORK_VECTOR);
-	exiting_irq();
 }
 
 void arch_irq_work_raise(void)
arch/x86/platform/uv/tlb_uv.c +1 -1
@@ -1272,7 +1272,7 @@ static void process_uv2_message(struct m
  * (the resource will not be freed until noninterruptable cpus see this
  *  interrupt; hardware may timeout the s/w ack and reply ERROR)
  */
-void uv_bau_message_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_uv_bau_message)
 {
 	int count = 0;
 	cycles_t time_start;
#91 lore raw

[tip: x86/entry] x86/entry: Convert various system vectors

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     41a5e11a9f35e97cc8872ffdb379319ea584f997
Gitweb:        https://git.kernel.org/tip/41a5e11a9f35e97cc8872ffdb379319ea584f997
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:41 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Convert various system vectors

Convert various system vectors to IDTENTRY_SYSVEC:

- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64-bit
- Remove the BUILD_INTERRUPT entries in 32-bit
- Remove the old prototypes

No functional change.

---
 arch/x86/entry/entry_64.S             | 19 +------------------
 arch/x86/include/asm/apic.h           | 13 +------------
 arch/x86/include/asm/entry_arch.h     | 25 +-----------------------
 arch/x86/include/asm/hw_irq.h         |  6 +------
 arch/x86/include/asm/idtentry.h       | 22 ++++++++++++++++++++-
 arch/x86/include/asm/irq_work.h       |  1 +-
 arch/x86/include/asm/traps.h          |  5 +-----
 arch/x86/include/asm/uv/uv_bau.h      |  8 +------
 arch/x86/kernel/cpu/mce/amd.c         |  5 ++---
 arch/x86/kernel/cpu/mce/therm_throt.c |  5 ++---
 arch/x86/kernel/cpu/mce/threshold.c   |  5 ++---
 arch/x86/kernel/idt.c                 | 28 +++++++++++++-------------
 arch/x86/kernel/irq_work.c            |  6 +++---
 arch/x86/platform/uv/tlb_uv.c         |  2 +-
 14 files changed, 48 insertions(+), 102 deletions(-)
arch/x86/entry/entry_64.S +0 -19
index f3ccb27..2301f62 100644
@@ -956,9 +956,6 @@ apicinterrupt3 \num \sym \do_sym
 POP_SECTION_IRQENTRY
 .endm
 
-#ifdef CONFIG_X86_UV
-apicinterrupt3 UV_BAU_MESSAGE			uv_bau_message_intr1		uv_bau_message_interrupt
-#endif
 
 #ifdef CONFIG_HAVE_KVM
 apicinterrupt3 POSTED_INTR_VECTOR		kvm_posted_intr_ipi		smp_kvm_posted_intr_ipi
@@ -966,26 +963,10 @@ apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR	kvm_posted_intr_wakeup_ipi	smp_kvm_post
 apicinterrupt3 POSTED_INTR_NESTED_VECTOR	kvm_posted_intr_nested_ipi	smp_kvm_posted_intr_nested_ipi
 #endif
 
-#ifdef CONFIG_X86_MCE_THRESHOLD
-apicinterrupt THRESHOLD_APIC_VECTOR		threshold_interrupt		smp_threshold_interrupt
-#endif
-
-#ifdef CONFIG_X86_MCE_AMD
-apicinterrupt DEFERRED_ERROR_VECTOR		deferred_error_interrupt	smp_deferred_error_interrupt
-#endif
-
-#ifdef CONFIG_X86_THERMAL_VECTOR
-apicinterrupt THERMAL_APIC_VECTOR		thermal_interrupt		smp_thermal_interrupt
-#endif
-
 #ifdef CONFIG_SMP
 apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
 
-#ifdef CONFIG_IRQ_WORK
-apicinterrupt IRQ_WORK_VECTOR			irq_work_interrupt		smp_irq_work_interrupt
-#endif
-
 /*
  * Reload gs selector with exception handling
  * edi:  new selector
arch/x86/include/asm/apic.h +0 -13
index 19e94af..a541686 100644
@@ -534,24 +534,11 @@ static inline void entering_ack_irq(void)
 	ack_APIC_irq();
 }
 
-static inline void ipi_entering_ack_irq(void)
-{
-	irq_enter();
-	ack_APIC_irq();
-	kvm_set_cpu_l1tf_flush_l1d();
-}
-
 static inline void exiting_irq(void)
 {
 	irq_exit();
 }
 
-static inline void exiting_ack_irq(void)
-{
-	ack_APIC_irq();
-	irq_exit();
-}
-
 extern void ioapic_zap_locks(void);
 
 #endif /* _ASM_X86_APIC_H */
arch/x86/include/asm/entry_arch.h +0 -25
index 2e2055b..69a5320 100644
@@ -20,28 +20,3 @@ BUILD_INTERRUPT(kvm_posted_intr_wakeup_ipi, POSTED_INTR_WAKEUP_VECTOR)
 BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)
 #endif
 
-/*
- * every pentium local APIC has two 'local interrupts', with a
- * soft-definable vector attached to both interrupts, one of
- * which is a timer interrupt, the other one is error counter
- * overflow. Linux uses the local APIC timer interrupt to get
- * a much simpler SMP time architecture:
- */
-#ifdef CONFIG_X86_LOCAL_APIC
-
-#ifdef CONFIG_IRQ_WORK
-BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR)
-#endif
-
-#ifdef CONFIG_X86_THERMAL_VECTOR
-BUILD_INTERRUPT(thermal_interrupt,THERMAL_APIC_VECTOR)
-#endif
-
-#ifdef CONFIG_X86_MCE_THRESHOLD
-BUILD_INTERRUPT(threshold_interrupt,THRESHOLD_APIC_VECTOR)
-#endif
-
-#ifdef CONFIG_X86_MCE_AMD
-BUILD_INTERRUPT(deferred_error_interrupt, DEFERRED_ERROR_VECTOR)
-#endif
-#endif
arch/x86/include/asm/hw_irq.h +0 -6
index 36a3869..7281c7e 100644
@@ -32,15 +32,9 @@
 extern asmlinkage void kvm_posted_intr_ipi(void);
 extern asmlinkage void kvm_posted_intr_wakeup_ipi(void);
 extern asmlinkage void kvm_posted_intr_nested_ipi(void);
-extern asmlinkage void irq_work_interrupt(void);
-extern asmlinkage void uv_bau_message_intr1(void);
 
-extern asmlinkage void thermal_interrupt(void);
 extern asmlinkage void reschedule_interrupt(void);
 
-extern asmlinkage void threshold_interrupt(void);
-extern asmlinkage void deferred_error_interrupt(void);
-
 #ifdef	CONFIG_X86_LOCAL_APIC
 struct irq_data;
 struct pci_dev;
arch/x86/include/asm/idtentry.h +22 -0
index 82cca1f..0eedfca 100644
@@ -584,6 +584,28 @@ DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,	sysvec_call_function_single
 DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_VECTOR,		sysvec_call_function);
 #endif
 
+#ifdef CONFIG_X86_LOCAL_APIC
+# ifdef CONFIG_X86_UV
+DECLARE_IDTENTRY_SYSVEC(UV_BAU_MESSAGE,			sysvec_uv_bau_message);
+# endif
+
+# ifdef CONFIG_X86_MCE_THRESHOLD
+DECLARE_IDTENTRY_SYSVEC(THRESHOLD_APIC_VECTOR,		sysvec_threshold);
+# endif
+
+# ifdef CONFIG_X86_MCE_AMD
+DECLARE_IDTENTRY_SYSVEC(DEFERRED_ERROR_VECTOR,		sysvec_deferred_error);
+# endif
+
+# ifdef CONFIG_X86_THERMAL_VECTOR
+DECLARE_IDTENTRY_SYSVEC(THERMAL_APIC_VECTOR,		sysvec_thermal);
+# endif
+
+# ifdef CONFIG_IRQ_WORK
+DECLARE_IDTENTRY_SYSVEC(IRQ_WORK_VECTOR,		sysvec_irq_work);
+# endif
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/irq_work.h +0 -1
index 80b35e3..800ffce 100644
@@ -10,7 +10,6 @@ static inline bool arch_irq_work_has_interrupt(void)
 	return boot_cpu_has(X86_FEATURE_APIC);
 }
 extern void arch_irq_work_raise(void);
-extern __visible void smp_irq_work_interrupt(struct pt_regs *regs);
 #else
 static inline bool arch_irq_work_has_interrupt(void)
 {
arch/x86/include/asm/traps.h +0 -5
index 0c40f37..714b1a3 100644
@@ -34,11 +34,6 @@ static inline int get_si_code(unsigned long condition)
 extern int panic_on_unrecovered_nmi;
 
 void math_emulate(struct math_emu_info *);
-#ifndef CONFIG_X86_32
-asmlinkage void smp_thermal_interrupt(struct pt_regs *regs);
-asmlinkage void smp_threshold_interrupt(struct pt_regs *regs);
-asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);
-#endif
 
 #ifdef CONFIG_VMAP_STACK
 void __noreturn handle_stack_overflow(const char *message,
arch/x86/include/asm/uv/uv_bau.h +2 -6
index 13687bf..f1188bd 100644
@@ -12,6 +12,8 @@
 #define _ASM_X86_UV_UV_BAU_H
 
 #include <linux/bitmap.h>
+#include <asm/idtentry.h>
+
 #define BITSPERBYTE 8
 
 /*
@@ -799,12 +801,6 @@ static inline void bau_cpubits_clear(struct bau_local_cpumask *dstp, int nbits)
 	bitmap_zero(&dstp->bits, nbits);
 }
 
-extern void uv_bau_message_intr1(void);
-#ifdef CONFIG_TRACING
-#define trace_uv_bau_message_intr1 uv_bau_message_intr1
-#endif
-extern void uv_bau_timeout_intr1(void);
-
 struct atomic_short {
 	short counter;
 };
arch/x86/kernel/cpu/mce/amd.c +2 -3
index 52de616..a906d68 100644
@@ -907,14 +907,13 @@ static void __log_error(unsigned int bank, u64 status, u64 addr, u64 misc)
 	mce_log(&m);
 }
 
-asmlinkage __visible void __irq_entry smp_deferred_error_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_deferred_error)
 {
-	entering_irq();
 	trace_deferred_error_apic_entry(DEFERRED_ERROR_VECTOR);
 	inc_irq_stat(irq_deferred_error_count);
 	deferred_error_int_vector();
 	trace_deferred_error_apic_exit(DEFERRED_ERROR_VECTOR);
-	exiting_ack_irq();
+	ack_APIC_irq();
 }
 
 /*
arch/x86/kernel/cpu/mce/therm_throt.c +2 -3
index f36dc07..a7cd2d2 100644
@@ -614,14 +614,13 @@ static void unexpected_thermal_interrupt(void)
 
 static void (*smp_thermal_vector)(void) = unexpected_thermal_interrupt;
 
-asmlinkage __visible void __irq_entry smp_thermal_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_thermal)
 {
-	entering_irq();
 	trace_thermal_apic_entry(THERMAL_APIC_VECTOR);
 	inc_irq_stat(irq_thermal_count);
 	smp_thermal_vector();
 	trace_thermal_apic_exit(THERMAL_APIC_VECTOR);
-	exiting_ack_irq();
+	ack_APIC_irq();
 }
 
 /* Thermal monitoring depends on APIC, ACPI and clock modulation */
arch/x86/kernel/cpu/mce/threshold.c +2 -3
index 28812cc..6a059a0 100644
@@ -21,12 +21,11 @@ static void default_threshold_interrupt(void)
 
 void (*mce_threshold_vector)(void) = default_threshold_interrupt;
 
-asmlinkage __visible void __irq_entry smp_threshold_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_threshold)
 {
-	entering_irq();
 	trace_threshold_apic_entry(THRESHOLD_APIC_VECTOR);
 	inc_irq_stat(irq_threshold_count);
 	mce_threshold_vector();
 	trace_threshold_apic_exit(THRESHOLD_APIC_VECTOR);
-	exiting_ack_irq();
+	ack_APIC_irq();
 }
arch/x86/kernel/idt.c +14 -14
index 018a542..3d811d0 100644
@@ -120,33 +120,33 @@ static const __initconst struct idt_data apic_idts[] = {
 #endif
 
 #ifdef CONFIG_X86_THERMAL_VECTOR
-	INTG(THERMAL_APIC_VECTOR,	thermal_interrupt),
+	INTG(THERMAL_APIC_VECTOR,		asm_sysvec_thermal),
 #endif
 
 #ifdef CONFIG_X86_MCE_THRESHOLD
-	INTG(THRESHOLD_APIC_VECTOR,	threshold_interrupt),
+	INTG(THRESHOLD_APIC_VECTOR,		asm_sysvec_threshold),
 #endif
 
 #ifdef CONFIG_X86_MCE_AMD
-	INTG(DEFERRED_ERROR_VECTOR,	deferred_error_interrupt),
+	INTG(DEFERRED_ERROR_VECTOR,		asm_sysvec_deferred_error),
 #endif
 
 #ifdef CONFIG_X86_LOCAL_APIC
-	INTG(LOCAL_TIMER_VECTOR,	asm_sysvec_apic_timer_interrupt),
-	INTG(X86_PLATFORM_IPI_VECTOR,	asm_sysvec_x86_platform_ipi),
+	INTG(LOCAL_TIMER_VECTOR,		asm_sysvec_apic_timer_interrupt),
+	INTG(X86_PLATFORM_IPI_VECTOR,		asm_sysvec_x86_platform_ipi),
 # ifdef CONFIG_HAVE_KVM
-	INTG(POSTED_INTR_VECTOR,	kvm_posted_intr_ipi),
-	INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi),
-	INTG(POSTED_INTR_NESTED_VECTOR, kvm_posted_intr_nested_ipi),
+	INTG(POSTED_INTR_VECTOR,		kvm_posted_intr_ipi),
+	INTG(POSTED_INTR_WAKEUP_VECTOR,		kvm_posted_intr_wakeup_ipi),
+	INTG(POSTED_INTR_NESTED_VECTOR,		kvm_posted_intr_nested_ipi),
 # endif
 # ifdef CONFIG_IRQ_WORK
-	INTG(IRQ_WORK_VECTOR,		irq_work_interrupt),
+	INTG(IRQ_WORK_VECTOR,			asm_sysvec_irq_work),
 # endif
-#ifdef CONFIG_X86_UV
-	INTG(UV_BAU_MESSAGE,		uv_bau_message_intr1),
-#endif
-	INTG(SPURIOUS_APIC_VECTOR,	asm_sysvec_spurious_apic_interrupt),
-	INTG(ERROR_APIC_VECTOR,		asm_sysvec_error_interrupt),
+# ifdef CONFIG_X86_UV
+	INTG(UV_BAU_MESSAGE,			asm_sysvec_uv_bau_message),
+# endif
+	INTG(SPURIOUS_APIC_VECTOR,		asm_sysvec_spurious_apic_interrupt),
+	INTG(ERROR_APIC_VECTOR,			asm_sysvec_error_interrupt),
 #endif
 };
 
arch/x86/kernel/irq_work.c +3 -3
index 80bee76..890d477 100644
@@ -9,18 +9,18 @@
 #include <linux/irq_work.h>
 #include <linux/hardirq.h>
 #include <asm/apic.h>
+#include <asm/idtentry.h>
 #include <asm/trace/irq_vectors.h>
 #include <linux/interrupt.h>
 
 #ifdef CONFIG_X86_LOCAL_APIC
-__visible void __irq_entry smp_irq_work_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_irq_work)
 {
-	ipi_entering_ack_irq();
+	ack_APIC_irq();
 	trace_irq_work_entry(IRQ_WORK_VECTOR);
 	inc_irq_stat(apic_irq_work_irqs);
 	irq_work_run();
 	trace_irq_work_exit(IRQ_WORK_VECTOR);
-	exiting_irq();
 }
 
 void arch_irq_work_raise(void)
arch/x86/platform/uv/tlb_uv.c +1 -1
index 1fd321f..b02e406 100644
@@ -1272,7 +1272,7 @@ static void process_uv2_message(struct msg_desc *mdp, struct bau_control *bcp)
  * (the resource will not be freed until noninterruptable cpus see this
  *  interrupt; hardware may timeout the s/w ack and reply ERROR)
  */
-void uv_bau_message_interrupt(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_uv_bau_message)
 {
 	int count = 0;
 	cycles_t time_start;
#92 lore raw

PATCH v9 29/39 x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC*

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Convert KVM specific system vectors to IDTENTRY_SYSVEC*:

The two empty stub handlers which only increment the stats counter do no need to run on the interrupt stack. Use IDTENTRY_SYSVEC_SIMPLE for them.

The wakeup handler does more work and runs on the interrupt stack.

None of these handlers need to save and restore the irq_regs pointer.

---
 arch/x86/entry/entry_64.S         |    7 -------
 arch/x86/include/asm/entry_arch.h |    7 -------
 arch/x86/include/asm/hw_irq.h     |    4 ----
 arch/x86/include/asm/idtentry.h   |    6 ++++++
 arch/x86/include/asm/irq.h        |    3 ---
 arch/x86/kernel/idt.c             |    6 +++---
 arch/x86/kernel/irq.c             |   24 ++++++------------------
 7 files changed, 15 insertions(+), 42 deletions(-)
arch/x86/entry/entry_64.S +0 -7
@@ -956,13 +956,6 @@ apicinterrupt3 \num \sym \do_sym
 POP_SECTION_IRQENTRY
 .endm
 
-
-#ifdef CONFIG_HAVE_KVM
-apicinterrupt3 POSTED_INTR_VECTOR		kvm_posted_intr_ipi		smp_kvm_posted_intr_ipi
-apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR	kvm_posted_intr_wakeup_ipi	smp_kvm_posted_intr_wakeup_ipi
-apicinterrupt3 POSTED_INTR_NESTED_VECTOR	kvm_posted_intr_nested_ipi	smp_kvm_posted_intr_nested_ipi
-#endif
-
 #ifdef CONFIG_SMP
 apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
arch/x86/include/asm/entry_arch.h +0 -7
@@ -13,10 +13,3 @@
 #ifdef CONFIG_SMP
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
 #endif
-
-#ifdef CONFIG_HAVE_KVM
-BUILD_INTERRUPT(kvm_posted_intr_ipi, POSTED_INTR_VECTOR)
-BUILD_INTERRUPT(kvm_posted_intr_wakeup_ipi, POSTED_INTR_WAKEUP_VECTOR)
-BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)
-#endif
-
arch/x86/include/asm/hw_irq.h +0 -4
@@ -29,10 +29,6 @@
 #include <asm/sections.h>
 
 /* Interrupt handlers registered during init_IRQ */
-extern asmlinkage void kvm_posted_intr_ipi(void);
-extern asmlinkage void kvm_posted_intr_wakeup_ipi(void);
-extern asmlinkage void kvm_posted_intr_nested_ipi(void);
-
 extern asmlinkage void reschedule_interrupt(void);
 
 #ifdef	CONFIG_X86_LOCAL_APIC
arch/x86/include/asm/idtentry.h +6 -0
@@ -602,6 +602,12 @@ DECLARE_IDTENTRY_SYSVEC(IRQ_WORK_VECTOR,
 # endif
 #endif
 
+#ifdef CONFIG_HAVE_KVM
+DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_VECTOR,		sysvec_kvm_posted_intr_ipi);
+DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_WAKEUP_VECTOR,	sysvec_kvm_posted_intr_wakeup_ipi);
+DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_NESTED_VECTOR,	sysvec_kvm_posted_intr_nested_ipi);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/irq.h +0 -3
@@ -26,9 +26,6 @@ extern void fixup_irqs(void);
 
 #ifdef CONFIG_HAVE_KVM
 extern void kvm_set_posted_intr_wakeup_handler(void (*handler)(void));
-extern __visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs);
-extern __visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs);
-extern __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs);
 #endif
 
 extern void (*x86_platform_ipi_callback)(void);
arch/x86/kernel/idt.c +3 -3
@@ -135,9 +135,9 @@ static const __initconst struct idt_data
 	INTG(LOCAL_TIMER_VECTOR,		asm_sysvec_apic_timer_interrupt),
 	INTG(X86_PLATFORM_IPI_VECTOR,		asm_sysvec_x86_platform_ipi),
 # ifdef CONFIG_HAVE_KVM
-	INTG(POSTED_INTR_VECTOR,		kvm_posted_intr_ipi),
-	INTG(POSTED_INTR_WAKEUP_VECTOR,		kvm_posted_intr_wakeup_ipi),
-	INTG(POSTED_INTR_NESTED_VECTOR,		kvm_posted_intr_nested_ipi),
+	INTG(POSTED_INTR_VECTOR,		asm_sysvec_kvm_posted_intr_ipi),
+	INTG(POSTED_INTR_WAKEUP_VECTOR,		asm_sysvec_kvm_posted_intr_wakeup_ipi),
+	INTG(POSTED_INTR_NESTED_VECTOR,		asm_sysvec_kvm_posted_intr_nested_ipi),
 # endif
 # ifdef CONFIG_IRQ_WORK
 	INTG(IRQ_WORK_VECTOR,			asm_sysvec_irq_work),
arch/x86/kernel/irq.c +6 -18
@@ -298,41 +298,29 @@ EXPORT_SYMBOL_GPL(kvm_set_posted_intr_wa
 /*
  * Handler for POSTED_INTERRUPT_VECTOR.
  */
-__visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_kvm_posted_intr_ipi)
 {
-	struct pt_regs *old_regs = set_irq_regs(regs);
-
-	entering_ack_irq();
+	ack_APIC_irq();
 	inc_irq_stat(kvm_posted_intr_ipis);
-	exiting_irq();
-	set_irq_regs(old_regs);
 }
 
 /*
  * Handler for POSTED_INTERRUPT_WAKEUP_VECTOR.
  */
-__visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_kvm_posted_intr_wakeup_ipi)
 {
-	struct pt_regs *old_regs = set_irq_regs(regs);
-
-	entering_ack_irq();
+	ack_APIC_irq();
 	inc_irq_stat(kvm_posted_intr_wakeup_ipis);
 	kvm_posted_intr_wakeup_handler();
-	exiting_irq();
-	set_irq_regs(old_regs);
 }
 
 /*
  * Handler for POSTED_INTERRUPT_NESTED_VECTOR.
  */
-__visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_kvm_posted_intr_nested_ipi)
 {
-	struct pt_regs *old_regs = set_irq_regs(regs);
-
-	entering_ack_irq();
+	ack_APIC_irq();
 	inc_irq_stat(kvm_posted_intr_nested_ipis);
-	exiting_irq();
-	set_irq_regs(old_regs);
 }
 #endif
 
#93 lore raw

[tip: x86/entry] x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC*

The following commit has been merged into the x86/entry branch of tip:

The following commit has been merged into the x86/entry branch of tip:

Commit-ID:     79e5785846099b7f48c92ffa299a4ddc1e4a98f6
Gitweb:        https://git.kernel.org/tip/79e5785846099b7f48c92ffa299a4ddc1e4a98f6
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Thu, 21 May 2020 22:05:42 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 26 May 2020 19:06:28 +02:00

x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC*

Convert KVM specific system vectors to IDTENTRY_SYSVEC*:

The two empty stub handlers which only increment the stats counter do no need to run on the interrupt stack. Use IDTENTRY_SYSVEC_SIMPLE for them.

The wakeup handler does more work and runs on the interrupt stack.

None of these handlers need to save and restore the irq_regs pointer.

---
 arch/x86/entry/entry_64.S         |  7 -------
 arch/x86/include/asm/entry_arch.h |  7 -------
 arch/x86/include/asm/hw_irq.h     |  4 ----
 arch/x86/include/asm/idtentry.h   |  6 ++++++
 arch/x86/include/asm/irq.h        |  3 ---
 arch/x86/kernel/idt.c             |  6 +++---
 arch/x86/kernel/irq.c             | 24 ++++++------------------
 7 files changed, 15 insertions(+), 42 deletions(-)
arch/x86/entry/entry_64.S +0 -7
index 2301f62..ea1f293 100644
@@ -956,13 +956,6 @@ apicinterrupt3 \num \sym \do_sym
 POP_SECTION_IRQENTRY
 .endm
 
-
-#ifdef CONFIG_HAVE_KVM
-apicinterrupt3 POSTED_INTR_VECTOR		kvm_posted_intr_ipi		smp_kvm_posted_intr_ipi
-apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR	kvm_posted_intr_wakeup_ipi	smp_kvm_posted_intr_wakeup_ipi
-apicinterrupt3 POSTED_INTR_NESTED_VECTOR	kvm_posted_intr_nested_ipi	smp_kvm_posted_intr_nested_ipi
-#endif
-
 #ifdef CONFIG_SMP
 apicinterrupt RESCHEDULE_VECTOR			reschedule_interrupt		smp_reschedule_interrupt
 #endif
arch/x86/include/asm/entry_arch.h +0 -7
index 69a5320..a01bb74 100644
@@ -13,10 +13,3 @@
 #ifdef CONFIG_SMP
 BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)
 #endif
-
-#ifdef CONFIG_HAVE_KVM
-BUILD_INTERRUPT(kvm_posted_intr_ipi, POSTED_INTR_VECTOR)
-BUILD_INTERRUPT(kvm_posted_intr_wakeup_ipi, POSTED_INTR_WAKEUP_VECTOR)
-BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)
-#endif
-
arch/x86/include/asm/hw_irq.h +0 -4
index 7281c7e..fd5e7c8 100644
@@ -29,10 +29,6 @@
 #include <asm/sections.h>
 
 /* Interrupt handlers registered during init_IRQ */
-extern asmlinkage void kvm_posted_intr_ipi(void);
-extern asmlinkage void kvm_posted_intr_wakeup_ipi(void);
-extern asmlinkage void kvm_posted_intr_nested_ipi(void);
-
 extern asmlinkage void reschedule_interrupt(void);
 
 #ifdef	CONFIG_X86_LOCAL_APIC
arch/x86/include/asm/idtentry.h +6 -0
index 0eedfca..7212eaf 100644
@@ -606,6 +606,12 @@ DECLARE_IDTENTRY_SYSVEC(IRQ_WORK_VECTOR,		sysvec_irq_work);
 # endif
 #endif
 
+#ifdef CONFIG_HAVE_KVM
+DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_VECTOR,		sysvec_kvm_posted_intr_ipi);
+DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_WAKEUP_VECTOR,	sysvec_kvm_posted_intr_wakeup_ipi);
+DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_NESTED_VECTOR,	sysvec_kvm_posted_intr_nested_ipi);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/irq.h +0 -3
index c7c43e8..f73dd3f 100644
@@ -26,9 +26,6 @@ extern void fixup_irqs(void);
 
 #ifdef CONFIG_HAVE_KVM
 extern void kvm_set_posted_intr_wakeup_handler(void (*handler)(void));
-extern __visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs);
-extern __visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs);
-extern __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs);
 #endif
 
 extern void (*x86_platform_ipi_callback)(void);
arch/x86/kernel/idt.c +3 -3
index 3d811d0..faaadd4 100644
@@ -135,9 +135,9 @@ static const __initconst struct idt_data apic_idts[] = {
 	INTG(LOCAL_TIMER_VECTOR,		asm_sysvec_apic_timer_interrupt),
 	INTG(X86_PLATFORM_IPI_VECTOR,		asm_sysvec_x86_platform_ipi),
 # ifdef CONFIG_HAVE_KVM
-	INTG(POSTED_INTR_VECTOR,		kvm_posted_intr_ipi),
-	INTG(POSTED_INTR_WAKEUP_VECTOR,		kvm_posted_intr_wakeup_ipi),
-	INTG(POSTED_INTR_NESTED_VECTOR,		kvm_posted_intr_nested_ipi),
+	INTG(POSTED_INTR_VECTOR,		asm_sysvec_kvm_posted_intr_ipi),
+	INTG(POSTED_INTR_WAKEUP_VECTOR,		asm_sysvec_kvm_posted_intr_wakeup_ipi),
+	INTG(POSTED_INTR_NESTED_VECTOR,		asm_sysvec_kvm_posted_intr_nested_ipi),
 # endif
 # ifdef CONFIG_IRQ_WORK
 	INTG(IRQ_WORK_VECTOR,			asm_sysvec_irq_work),
arch/x86/kernel/irq.c +6 -18
index 7e30052..1810602 100644
@@ -298,41 +298,29 @@ EXPORT_SYMBOL_GPL(kvm_set_posted_intr_wakeup_handler);
 /*
  * Handler for POSTED_INTERRUPT_VECTOR.
  */
-__visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_kvm_posted_intr_ipi)
 {
-	struct pt_regs *old_regs = set_irq_regs(regs);
-
-	entering_ack_irq();
+	ack_APIC_irq();
 	inc_irq_stat(kvm_posted_intr_ipis);
-	exiting_irq();
-	set_irq_regs(old_regs);
 }
 
 /*
  * Handler for POSTED_INTERRUPT_WAKEUP_VECTOR.
  */
-__visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_kvm_posted_intr_wakeup_ipi)
 {
-	struct pt_regs *old_regs = set_irq_regs(regs);
-
-	entering_ack_irq();
+	ack_APIC_irq();
 	inc_irq_stat(kvm_posted_intr_wakeup_ipis);
 	kvm_posted_intr_wakeup_handler();
-	exiting_irq();
-	set_irq_regs(old_regs);
 }
 
 /*
  * Handler for POSTED_INTERRUPT_NESTED_VECTOR.
  */
-__visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_kvm_posted_intr_nested_ipi)
 {
-	struct pt_regs *old_regs = set_irq_regs(regs);
-
-	entering_ack_irq();
+	ack_APIC_irq();
 	inc_irq_stat(kvm_posted_intr_nested_ipis);
-	exiting_irq();
-	set_irq_regs(old_regs);
 }
 #endif
 
#94 lore raw

PATCH v9 30/39 x86/entry: Convert various hypervisor vectors to IDTENTRY_SYSVEC

In reply to Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Convert various hypervisor vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes

No functional change.

---
 arch/x86/entry/entry_32.S       |   14 --------------
 arch/x86/entry/entry_64.S       |   17 -----------------
 arch/x86/hyperv/hv_init.c       |    9 +++------
 arch/x86/include/asm/acrn.h     |   11 -----------
 arch/x86/include/asm/apic.h     |   20 --------------------
 arch/x86/include/asm/idtentry.h |   10 ++++++++++
 arch/x86/include/asm/mshyperv.h |   13 -------------
 arch/x86/kernel/cpu/acrn.c      |    9 ++++-----
 arch/x86/kernel/cpu/mshyperv.c  |   22 ++++++++++------------
 9 files changed, 27 insertions(+), 98 deletions(-)
arch/x86/entry/entry_32.S +0 -14
@@ -1342,20 +1342,6 @@ BUILD_INTERRUPT3(xen_hvm_callback_vector
 		 xen_evtchn_do_upcall)
 #endif
 
-
-#if IS_ENABLED(CONFIG_HYPERV)
-
-BUILD_INTERRUPT3(hyperv_callback_vector, HYPERVISOR_CALLBACK_VECTOR,
-		 hyperv_vector_handler)
-
-BUILD_INTERRUPT3(hyperv_reenlightenment_vector, HYPERV_REENLIGHTENMENT_VECTOR,
-		 hyperv_reenlightenment_intr)
-
-BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR,
-		 hv_stimer0_vector_handler)
-
-#endif /* CONFIG_HYPERV */
-
 SYM_CODE_START_LOCAL_NOALIGN(handle_exception)
 	/* the function address is in %gs's slot on the stack */
 	SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -17
@@ -1116,23 +1116,6 @@ apicinterrupt3 HYPERVISOR_CALLBACK_VECTO
 	xen_hvm_callback_vector xen_evtchn_do_upcall
 #endif
 
-
-#if IS_ENABLED(CONFIG_HYPERV)
-apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \
-	hyperv_callback_vector hyperv_vector_handler
-
-apicinterrupt3 HYPERV_REENLIGHTENMENT_VECTOR \
-	hyperv_reenlightenment_vector hyperv_reenlightenment_intr
-
-apicinterrupt3 HYPERV_STIMER0_VECTOR \
-	hv_stimer0_callback_vector hv_stimer0_vector_handler
-#endif /* CONFIG_HYPERV */
-
-#if IS_ENABLED(CONFIG_ACRN_GUEST)
-apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \
-	acrn_hv_callback_vector acrn_hv_vector_handler
-#endif
-
 /*
  * Save all registers in pt_regs, and switch gs if needed.
  * Use slow, but surefire "are we in kernel?" check.
arch/x86/hyperv/hv_init.c +3 -6
@@ -15,6 +15,7 @@
 #include <asm/hypervisor.h>
 #include <asm/hyperv-tlfs.h>
 #include <asm/mshyperv.h>
+#include <asm/idtentry.h>
 #include <linux/version.h>
 #include <linux/vmalloc.h>
 #include <linux/mm.h>
@@ -153,15 +154,11 @@ static inline bool hv_reenlightenment_av
 		ms_hyperv.features & HV_X64_ACCESS_REENLIGHTENMENT;
 }
 
-__visible void __irq_entry hyperv_reenlightenment_intr(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_reenlightenment)
 {
-	entering_ack_irq();
-
+	ack_APIC_irq();
 	inc_irq_stat(irq_hv_reenlightenment_count);
-
 	schedule_delayed_work(&hv_reenlightenment_work, HZ/10);
-
-	exiting_irq();
 }
 
 void set_hv_tscchange_cb(void (*cb)(void))
arch/x86/include/asm/acrn.h deleted +0 -11
@@ -1,11 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_X86_ACRN_H
-#define _ASM_X86_ACRN_H
-
-extern void acrn_hv_callback_vector(void);
-#ifdef CONFIG_TRACING
-#define trace_acrn_hv_callback_vector acrn_hv_callback_vector
-#endif
-
-extern void acrn_hv_vector_handler(struct pt_regs *regs);
-#endif /* _ASM_X86_ACRN_H */
arch/x86/include/asm/apic.h +0 -20
@@ -519,26 +519,6 @@ static inline bool apic_id_is_primary_th
 static inline void apic_smt_update(void) { }
 #endif
 
-extern void irq_enter(void);
-extern void irq_exit(void);
-
-static inline void entering_irq(void)
-{
-	irq_enter();
-	kvm_set_cpu_l1tf_flush_l1d();
-}
-
-static inline void entering_ack_irq(void)
-{
-	entering_irq();
-	ack_APIC_irq();
-}
-
-static inline void exiting_irq(void)
-{
-	irq_exit();
-}
-
 extern void ioapic_zap_locks(void);
 
 #endif /* _ASM_X86_APIC_H */
arch/x86/include/asm/idtentry.h +10 -0
@@ -608,6 +608,16 @@ DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_WAKE
 DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_NESTED_VECTOR,	sysvec_kvm_posted_intr_nested_ipi);
 #endif
 
+#if IS_ENABLED(CONFIG_HYPERV)
+DECLARE_IDTENTRY_SYSVEC(HYPERVISOR_CALLBACK_VECTOR,	sysvec_hyperv_callback);
+DECLARE_IDTENTRY_SYSVEC(HYPERVISOR_REENLIGHTENMENT_VECTOR,	sysvec_hyperv_reenlightenment);
+DECLARE_IDTENTRY_SYSVEC(HYPERVISOR_STIMER0_VECTOR,	sysvec_hyperv_stimer0);
+#endif
+
+#if IS_ENABLED(CONFIG_ACRN_GUEST)
+DECLARE_IDTENTRY_SYSVEC(HYPERVISOR_CALLBACK_VECTOR,	sysvec_acrn_hv_callback);
+#endif
+
 #undef X86_TRAP_OTHER
 
 #endif
arch/x86/include/asm/mshyperv.h +0 -13
@@ -54,20 +54,8 @@ typedef int (*hyperv_fill_flush_list_fun
 	vclocks_set_used(VDSO_CLOCKMODE_HVCLOCK);
 #define hv_get_raw_timer() rdtsc_ordered()
 
-void hyperv_callback_vector(void);
-void hyperv_reenlightenment_vector(void);
-#ifdef CONFIG_TRACING
-#define trace_hyperv_callback_vector hyperv_callback_vector
-#endif
 void hyperv_vector_handler(struct pt_regs *regs);
 
-/*
- * Routines for stimer0 Direct Mode handling.
- * On x86/x64, there are no percpu actions to take.
- */
-void hv_stimer0_vector_handler(struct pt_regs *regs);
-void hv_stimer0_callback_vector(void);
-
 static inline void hv_enable_stimer0_percpu_irq(int irq) {}
 static inline void hv_disable_stimer0_percpu_irq(int irq) {}
 
@@ -226,7 +214,6 @@ void hyperv_setup_mmu_ops(void);
 void *hv_alloc_hyperv_page(void);
 void *hv_alloc_hyperv_zeroed_page(void);
 void hv_free_hyperv_page(unsigned long addr);
-void hyperv_reenlightenment_intr(struct pt_regs *regs);
 void set_hv_tscchange_cb(void (*cb)(void));
 void clear_hv_tscchange_cb(void);
 void hyperv_stop_tsc_emulation(void);
arch/x86/kernel/cpu/acrn.c +4 -5
@@ -10,10 +10,10 @@
  */
 
 #include <linux/interrupt.h>
-#include <asm/acrn.h>
 #include <asm/apic.h>
 #include <asm/desc.h>
 #include <asm/hypervisor.h>
+#include <asm/idtentry.h>
 #include <asm/irq_regs.h>
 
 static uint32_t __init acrn_detect(void)
@@ -24,7 +24,7 @@ static uint32_t __init acrn_detect(void)
 static void __init acrn_init_platform(void)
 {
 	/* Setup the IDT for ACRN hypervisor callback */
-	alloc_intr_gate(HYPERVISOR_CALLBACK_VECTOR, acrn_hv_callback_vector);
+	alloc_intr_gate(HYPERVISOR_CALLBACK_VECTOR, asm_sysvec_acrn_hv_callback);
 }
 
 static bool acrn_x2apic_available(void)
@@ -39,7 +39,7 @@ static bool acrn_x2apic_available(void)
 
 static void (*acrn_intr_handler)(void);
 
-__visible void __irq_entry acrn_hv_vector_handler(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_acrn_hv_callback)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
@@ -50,13 +50,12 @@ static void (*acrn_intr_handler)(void);
 	 * will block the interrupt whose vector is lower than
 	 * HYPERVISOR_CALLBACK_VECTOR.
 	 */
-	entering_ack_irq();
+	ack_APIC_irq();
 	inc_irq_stat(irq_hv_callback_count);
 
 	if (acrn_intr_handler)
 		acrn_intr_handler();
 
-	exiting_irq();
 	set_irq_regs(old_regs);
 }
 
arch/x86/kernel/cpu/mshyperv.c +10 -12
@@ -23,6 +23,7 @@
 #include <asm/hyperv-tlfs.h>
 #include <asm/mshyperv.h>
 #include <asm/desc.h>
+#include <asm/idtentry.h>
 #include <asm/irq_regs.h>
 #include <asm/i8259.h>
 #include <asm/apic.h>
@@ -40,11 +41,10 @@ static void (*hv_stimer0_handler)(void);
 static void (*hv_kexec_handler)(void);
 static void (*hv_crash_handler)(struct pt_regs *regs);
 
-__visible void __irq_entry hyperv_vector_handler(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
-	entering_irq();
 	inc_irq_stat(irq_hv_callback_count);
 	if (vmbus_handler)
 		vmbus_handler();
@@ -52,7 +52,6 @@ static void (*hv_crash_handler)(struct p
 	if (ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED)
 		ack_APIC_irq();
 
-	exiting_irq();
 	set_irq_regs(old_regs);
 }
 
@@ -73,19 +72,16 @@ EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);
  * Routines to do per-architecture handling of stimer0
  * interrupts when in Direct Mode
  */
-
-__visible void __irq_entry hv_stimer0_vector_handler(struct pt_regs *regs)
+DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_stimer0)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
 
-	entering_irq();
 	inc_irq_stat(hyperv_stimer0_count);
 	if (hv_stimer0_handler)
 		hv_stimer0_handler();
 	add_interrupt_randomness(HYPERV_STIMER0_VECTOR, 0);
 	ack_APIC_irq();
 
-	exiting_irq();
 	set_irq_regs(old_regs);
 }
 
@@ -331,17 +327,19 @@ static void __init ms_hyperv_init_platfo
 	x86_platform.apic_post_init = hyperv_init;
 	hyperv_setup_mmu_ops();
 	/* Setup the IDT for hypervisor callback */
-	alloc_intr_gate(HYPERVISOR_CALLBACK_VECTOR, hyperv_callback_vector);
+	alloc_intr_gate(HYPERVISOR_CALLBACK_VECTOR, sysvec_hyperv_callback);
 
 	/* Setup the IDT for reenlightenment notifications */
-	if (ms_hyperv.features & HV_X64_ACCESS_REENLIGHTENMENT)
+	if (ms_hyperv.features & HV_X64_ACCESS_REENLIGHTENMENT) {
 		alloc_intr_gate(HYPERV_REENLIGHTENMENT_VECTOR,
-				hyperv_reenlightenment_vector);
+				asm_sysvec_hyperv_reenlightenment);
+	}
 
 	/* Setup the IDT for stimer0 */
-	if (ms_hyperv.misc_features & HV_STIMER_DIRECT_MODE_AVAILABLE)
+	if (ms_hyperv.misc_features & HV_STIMER_DIRECT_MODE_AVAILABLE) {
 		alloc_intr_gate(HYPERV_STIMER0_VECTOR,
-				hv_stimer0_callback_vector);
+				asm_sysvec_hyperv_stimer0);
+	}
 
 # ifdef CONFIG_SMP
 	smp_ops.smp_prepare_boot_cpu = hv_smp_prepare_boot_cpu;
maintainer Reviewed-by on v9 30/39
#95 lore raw

Re: [patch V9 30/39] x86/entry: Convert various hypervisor vectors to IDTENTRY_SYSVEC

Quoted 9 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>

Convert various hypervisor vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes

Quoted 4 lines

No functional change.

For Hyper-V bits: