x86/entry: Rework leftovers and merge plan
Subsystems (14)
v6 cover x86/entry: Rework leftovers and merge plan
Folks!
This is V6 of the rework series. V5 can be found here:
https://lore.kernel.org/r/[email protected]
The V6 leftover series is based on:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-base-v6
which is the reworked base series from part 1-4 of the original 5 part series with a few changes which are described in detail below in the merge plan section.
V6 has the following changes vs. V5:
- Rebased on top entry-base-v6
- Addressed Stevens request to split up the hardware latency detector. This are 3 patches now as I couldn't resist to cleanup the timestamping mess in that code before splitting it up.
- Dropped the KVM/SVM change as that is going to be routed differently. See below.
The full series is available from:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-v6-the-rest
On top of that the kvm changes are applied for completeness and available from:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-v6-full
Merge plan:
-----------
After figuring out that the entry pile and next are not really happy with each other, I spent quite some time to come up with a plan.
The goal was to:
- not let Stephen Rothwell grow more grey hair when trying to resolve the conflicts
- allow the affected trees (RCU and KVM) to take a small part of the series into their trees while making sure that the x86/entry branch is complete and contains the required RCU and KVM changes as well.
About 10 hours of patch tetris later the solution looks like this:
I've reshuffled the patches so that they are grouped by subsystem instead of having the cross tree/subsystem patches close to the actual usage site in the x86/entry series.
This allowed me to tag these subsytem parts and they contain just the minimal subset of changes to be able to build and boot.
The resulting tag list is:
- noinstr-lds-2020-05-15
A single commit containing the vmlinux.lds.h change which introduces the noinstr.text section.
- noinstr-core-2020-05-15
Based on noinstr-lds-2020-05-15 and contains the core changes
- noinstr-core-for-kvm-2020-05-15
Subset of noinstr-core-2020-05-15 which is required to let KVM pull the KVM async pagefault cleanup and base the guest_enter/exit() and noinstr changes on top.
- noinstr-rcu-nmi-2020-05-15
Based on the core/rcu branch in the tip tree. It has merged in noinstr-lds-2020-05-15 and contains the nmi_enter/exit() changes along with the noinstr section changes on top.
This tag is intended to be pulled by Paul into his rcu/next branch so he can sort the conflicts and base further work on top.
- noinstr-core-2020-05-15
Based on noinstr-core-for-kvm-2020-05-15 and contains the async page fault cleanup which goes into x86/entry so the IDTENTRY conversion of #PF which also touches the async pagefault code can be applied on top
This tag is intended to be pulled by Paolo into his next branch so he can work against these changes and the merged result is also target for the rebased version of the KVM guest_enter/exit() changes. These are not part of the entry-v6-base tag. I'm going to post them as a separate series because the original ones are conflicting with work in that area in the KVM tree.
- noinstr-kcsan-2020-05015, noinstr-kprobes-2020-05-15, noinstr-objtool-2020-05-15
TIP tree internal tags which I added to reduce the brain-melt.
The x86/entry branch is based on the TIP x86/entry branch and has the following branches and tags merged and patches from part 1-4 applied:
- x86/asm because this has conflicting changes vs. #DF
- A small set of preparatory changes and fixes which are independent of the noinstr mechanics
- noinstr-objtool-2020-05-15
- noinstr-core-2020-05-15
- noinstr-kprobes-2020-05-15
- noinstr-rcu-nmi-2020-05-15
- noinstr-kcsan-2020-05015
- noinstr-x86-kvm-2020-05-15
- The part 1-4 patches up to
51336ff8b658 ("x86/entry: Convert double fault exception to IDTENTRY_DF")
This is tagged as entry-v6-base
The remaining patches in this leftover series will be applied on top.
If this works for all maintainers involved, then I'm going to pull the tags and branches into the tip-tree which makes them immutable.
If not, please tell me ASAP that I should restart the patch tetris session after hiding in a brown paperbag for some time to recover from brain melt.
Thanks,
tglx
PATCH v6 1/37 tracing/hwlat: Use ktime_get_mono_fast_ns()
Timestamping in the hardware latency detector uses sched_clock() underneath and depends on CONFIG_GENERIC_SCHED_CLOCK=n because sched clocks from that subsystem are not NMI safe.
ktime_get_mono_fast_ns() is NMI safe and available on all architectures.
Replace the time getter, get rid of the CONFIG_GENERIC_SCHED_CLOCK=n dependency and cleanup the horrible macro maze which encapsulates u64 math in u64 macros.
--- kernel/trace/trace_hwlat.c | 59 +++++++++++++++++++-------------------------- 1 file changed, 25 insertions(+), 34 deletions(-)
kernel/trace/trace_hwlat.c +25 -34
@@ -131,29 +131,19 @@ static void trace_hwlat_sample(struct hwtrace_buffer_unlock_commit_nostack (buffer, event); } -/* Macros to encapsulate the time capturing infrastructure */ -#define time_type u64 -#define time_get () trace_clock_local() -#define time_to_us (x) div_u64(x, 1000) -#define time_sub (a, b) ((a) - (b)) -#define init_time (a, b) (a = b) -#define time_u64 (a) a - +/* +* Timestamping uses ktime_get_mono_fast(), the NMI safe access to +* CLOCK_MONOTONIC. +*/ void trace_hwlat_callback (bool enter) {if (smp_processor_id () != nmi_cpu)return ; -/* -* Currently trace_clock_local() calls sched_clock() and the -* generic version is not NMI safe. -*/ -if (!IS_ENABLED (CONFIG_GENERIC_SCHED_CLOCK )) { -if (enter) - nmi_ts_start =time_get (); -else - nmi_total_ts +=time_get () - nmi_ts_start; - } +if (enter) + nmi_ts_start =ktime_get_mono_fast_ns (); +else + nmi_total_ts +=ktime_get_mono_fast_ns () - nmi_ts_start;if (enter) nmi_count++;
@@ -165,20 +155,22 @@ void trace_hwlat_callback(bool enter)* Used to repeatedly capture the CPU TSC (or similar), looking for potential * hardware-induced latency. Called with interrupts disabled and with * hwlat_data.lock held. +* +* Use ktime_get_mono_fast() here as well because it does not wait on the +* timekeeping seqcount like ktime_get_mono(). */ static int get_sample (void ) {struct trace_array *tr = hwlat_trace;struct hwlat_sample s; -time_type start, t1, t2, last_t2; +u64 start, t1, t2, last_t2, thresh;s64 diff, outer_diff, total, last_total =0 ;u64 sample =0 ; -u64 thresh = tracing_thresh;u64 outer_sample =0 ;int ret =-1 ;unsigned int count =0 ; -do_div (thresh,NSEC_PER_USEC );/* modifies interval value */ + thresh =div_u64 (tracing_thresh,NSEC_PER_USEC ); nmi_cpu =smp_processor_id (); nmi_total_ts =0 ;
@@ -188,18 +180,20 @@ static int get_sample(void) trace_hwlat_callback_enabled = true; -init_time (last_t2,0 ); - start =time_get ();/* start timestamp */ +/* start timestamp */ + start =ktime_get_mono_fast_ns (); outer_diff =0 ; + last_t2 =0 ; do { - t1 = time_get(); /* we'll look for a discontinuity */ - t2 = time_get(); + /* we'll look for a discontinuity */ + t1 = ktime_get_mono_fast_ns(); + t2 = ktime_get_mono_fast_ns(); - if (time_u64(last_t2)) { + if (last_t2) { /* Check the delta from outer loop (t2 to next t1) */ - outer_diff = time_to_us(time_sub(t1, last_t2)); + outer_diff = div_u64(t1 - last_t2, NSEC_PER_USEC); /* This shouldn't happen */ if (outer_diff < 0) { pr_err(BANNER "time running backwards\n");
@@ -210,7 +204,8 @@ static int get_sample(void) } last_t2 = t2; - total =time_to_us (time_sub (t2, start));/* sample width */ +/* sample width */ + total =div_u64 (t2 - start,NSEC_PER_USEC );/* Check for possible overflows */ if (total < last_total) {
@@ -220,7 +215,7 @@ static int get_sample(void) last_total = total;/* This checks the inner loop (t1 to t2) */ - diff =time_to_us (time_sub (t2, t1));/* current diff */ + diff =div_u64 (t2 - t1,NSEC_PER_USEC );if (diff > thresh || outer_diff > thresh) {if (!count)
@@ -251,15 +246,11 @@ static int get_sample(void) ret =1 ; -/* We read in microseconds */ -if (nmi_total_ts) -do_div (nmi_total_ts,NSEC_PER_USEC ); - hwlat_data.count++; s.seqnum = hwlat_data.count; s.duration = sample; s.outer_duration = outer_sample; - s.nmi_total_ts = nmi_total_ts; + s.nmi_total_ts =div_u64 (nmi_total_ts,NSEC_PER_USEC ); s.nmi_count = nmi_count; s.count = count;trace_hwlat_sample (&s);
Steven Rostedt
maintainerRe: [patch V6 01/37] tracing/hwlat: Use ktime_get_mono_fast_ns()
Quoted 9 lines · On Sat, 16 May 2020 01:45:48 +0200 Thomas Gleixner <[email protected]> wrote:
Timestamping in the hardware latency detector uses
sched_clock()underneath and depends onCONFIG_GENERIC_SCHED_CLOCK=nbecause sched clocks from that subsystem are not NMI safe.
ktime_get_mono_fast_ns()is NMI safe and available on all architectures.Replace the time getter, get rid of the
CONFIG_GENERIC_SCHED_CLOCK=ndependency and cleanup the horrible macro maze which encapsulates u64 math in u64 macros.
Good riddance. That macro maze was due to supporting the same code upstream as we had in RHEL RT, where the math and time keeping functions available between the two kernels were different.
That's been dealt with, but the macros never got cleaned up.
Quoted 47 lines
Signed-off-by Thomas Gleixner <[email protected]>--- kernel/trace/trace_hwlat.c | 59 +++++++++++++++++++-------------------------- 1 file changed, 25 insertions(+), 34 deletions(-)kernel/trace/trace_hwlat.c +8
-18@@ -131,29 +131,19 @@ static void trace_hwlat_sample(struct hwtrace_buffer_unlock_commit_nostack (buffer, event); } -/* Macros to encapsulate the time capturing infrastructure */ -#define time_type u64 -#define time_get () trace_clock_local() -#define time_to_us (x) div_u64(x, 1000) -#define time_sub (a, b) ((a) - (b)) -#define init_time (a, b) (a = b) -#define time_u64 (a) a - +/* +* Timestamping uses ktime_get_mono_fast(), the NMI safe access to +* CLOCK_MONOTONIC. +*/ void trace_hwlat_callback (bool enter) {if (smp_processor_id () != nmi_cpu)return ; -/* -* Currently trace_clock_local() calls sched_clock() and the -* generic version is not NMI safe. -*/ -if (!IS_ENABLED (CONFIG_GENERIC_SCHED_CLOCK )) { -if (enter) - nmi_ts_start =time_get (); -else - nmi_total_ts +=time_get () - nmi_ts_start; - } +if (enter) + nmi_ts_start =ktime_get_mono_fast_ns (); +else + nmi_total_ts +=ktime_get_mono_fast_ns () - nmi_ts_start;if (enter) nmi_count++;
kernel/trace/trace_hwlat.c void trace_hwlat_callback(bool enter) · line 160
Quoted 6 lines
kernel/trace/trace_hwlat.c +3
-0@@ -165,20 +155,22 @@ void trace_hwlat_callback(bool enter)* Used to repeatedly capture the CPU TSC (or similar), looking for potential * hardware-induced latency. Called with interrupts disabled and with * hwlat_data.lock held. +* +* Use ktime_get_mono_fast() here as well because it does not wait on the +* timekeeping seqcount like ktime_get_mono().
When doing a "git grep ktime_get_mono" I only find ktime_get_mono_fast_ns() (and this comment), so I don't know what to compare that to. Did you mean another function?
The rest looks fine (although, I see other things I need to clean up in this code ;-)
-- Steve
Quoted 14 lines
kernel/trace/trace_hwlat.c +1
-2*/ static int get_sample (void ) {struct trace_array *tr = hwlat_trace;struct hwlat_sample s; -time_type start, t1, t2, last_t2; +u64 start, t1, t2, last_t2, thresh;s64 diff, outer_diff, total, last_total =0 ;u64 sample =0 ; -u64 thresh = tracing_thresh;u64 outer_sample =0 ;int ret =-1 ;unsigned int count =0 ;
Re: [patch V6 01/37] tracing/hwlat: Use ktime_get_mono_fast_ns()
Quoted 7 lines · Steven Rostedt
Quoted 5 lines · On Sat, 16 May 2020 01:45:48 +0200 Thomas Gleixner <[email protected]> wrote:
kernel/trace/trace_hwlat.c +4
-0+if (enter) + nmi_ts_start =ktime_get_mono_fast_ns (); +else + nmi_total_ts +=ktime_get_mono_fast_ns () - nmi_ts_start;
Quoted 13 lines
kernel/trace/trace_hwlat.c void trace_hwlat_callback(bool enter) · line 160
Quoted 9 lines
kernel/trace/trace_hwlat.c +3
-0if (enter) nmi_count++;@@ -165,20 +155,22 @@ void trace_hwlat_callback(bool enter)* Used to repeatedly capture the CPU TSC (or similar), looking for potential * hardware-induced latency. Called with interrupts disabled and with * hwlat_data.lock held. +* +* Use ktime_get_mono_fast() here as well because it does not wait on the +* timekeeping seqcount like ktime_get_mono(). When doing a "git grep
ktime_get_mono" I only findktime_get_mono_fast_ns()(and this comment), so I don't know what to compare that to. Did you mean another function?
Yeah. I fatfingered the comment. The code uses ktime_get_mono_fast_ns().
Quoted 2 lines
The rest looks fine (although, I see other things I need to clean up in this code ;-)
Quite some ...
Thanks,
tglx
Steven Rostedt
maintainerRe: [patch V6 01/37] tracing/hwlat: Use ktime_get_mono_fast_ns()
Quoted 12 lines · On Tue, 19 May 2020 23:45:10 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 11 lines
kernel/trace/trace_hwlat.c void trace_hwlat_callback(bool enter) · line 160
Quoted 7 lines
kernel/trace/trace_hwlat.c +3
-0@@ -165,20 +155,22 @@ void trace_hwlat_callback(bool enter)* Used to repeatedly capture the CPU TSC (or similar), looking for potential * hardware-induced latency. Called with interrupts disabled and with * hwlat_data.lock held. +* +* Use ktime_get_mono_fast() here as well because it does not wait on the +* timekeeping seqcount like ktime_get_mono(). When doing a "git grep
ktime_get_mono" I only findktime_get_mono_fast_ns()(and this comment), so I don't know what to compare that to. Did you mean another function?
Quoted 1 line
Yeah. I fatfingered the comment. The code uses
ktime_get_mono_fast_ns().
Well, I assumed that's what you meant with "ktime_get_mono_fast()" but I don't know what function you are comparing it to that waits on the seqcount like "ktime_get_mono()" as there is no such function.
-- Steve
Re: [patch V6 01/37] tracing/hwlat: Use ktime_get_mono_fast_ns()
Quoted 15 lines · Steven Rostedt
Quoted 12 lines · On Tue, 19 May 2020 23:45:10 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 11 lines
kernel/trace/trace_hwlat.c void trace_hwlat_callback(bool enter) · line 160
Quoted 7 lines
kernel/trace/trace_hwlat.c +3
-0@@ -165,20 +155,22 @@ void trace_hwlat_callback(bool enter)* Used to repeatedly capture the CPU TSC (or similar), looking for potential * hardware-induced latency. Called with interrupts disabled and with * hwlat_data.lock held. +* +* Use ktime_get_mono_fast() here as well because it does not wait on the +* timekeeping seqcount like ktime_get_mono(). When doing a "git grep
ktime_get_mono" I only findktime_get_mono_fast_ns()(and this comment), so I don't know what to compare that to. Did you mean another function?
Quoted 5 lines
Quoted 1 line
Yeah. I fatfingered the comment. The code uses
ktime_get_mono_fast_ns().Well, I assumed that's what you meant with "
ktime_get_mono_fast()" but I don't know what function you are comparing it to that waits on the seqcount like "ktime_get_mono()" as there is no such function.
Gah. ktime_get_mono_fast_ns() and ktime_get() of course.
Peter Zijlstra
maintainerRe: [patch V6 01/37] tracing/hwlat: Use ktime_get_mono_fast_ns()
In reply to Thomas GleixnerQuoted 3 lines · Thomas Gleixner
Timestamping in the hardware latency detector uses
sched_clock()underneath and depends onCONFIG_GENERIC_SCHED_CLOCK=nbecause sched clocks from that subsystem are not NMI safe.
AFAICT that's not actually true, see commit:
1809bfa44e10 ("timers, sched/clock: Avoid deadlock during read from NMI")
that said, no objection to switching to ktime, people that run HPET clocks deserve all the pain and suffering available with those setups.
Re: [patch V6 01/37] tracing/hwlat: Use ktime_get_mono_fast_ns()
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Thomas Gleixner
Timestamping in the hardware latency detector uses
sched_clock()underneath and depends onCONFIG_GENERIC_SCHED_CLOCK=nbecause sched clocks from that subsystem are not NMI safe.AFAICT that's not actually true, see commit:
1809bfa44e10("timers, sched/clock: Avoid deadlock during read from NMI")that said, no objection to switching to ktime, people that run HPET clocks deserve all the pain and suffering available with those setups.
Correct ...
PATCH v6 2/37 tracing/hwlat: Split ftrace_nmi_enter/exit()
In reply to Thomas GleixnerThe hardware latency tracer calls into timekeeping and ends up in various instrumentable functions which is problematic vs. the kprobe handling especially the text poke machinery. It's invoked from nmi_enter/exit(), i.e. non-instrumentable code.
Split it into two parts:
1) NMI counter, only invoked on nmi_enter() and noinstr safe
2) NMI timestamping, to be invoked from instrumentable code
Move it into the rcu is watching regions of nmi_enter/exit() even if there is no actual RCU dependency right now but there is also no point in having it early.
The actual split of nmi_enter/exit() is done in a separate step.
--- include/linux/ftrace_irq.h | 31 +++++++++++++++++++------------ include/linux/hardirq.h | 5 +++-- kernel/trace/trace_hwlat.c | 19 ++++++++++++------- 3 files changed, 34 insertions(+), 21 deletions(-)
include/linux/ftrace_irq.h +19 -12
@@ -4,23 +4,30 @@#ifdef CONFIG_HWLAT_TRACER extern bool trace_hwlat_callback_enabled; -extern void trace_hwlat_callback (bool enter); -#endif +extern void trace_hwlat_count_nmi (void ); +extern void trace_hwlat_timestamp (bool enter); -static inline void ftrace_nmi_enter (void ) +static __always_inline void ftrace_count_nmi(void) { -#ifdef CONFIG_HWLAT_TRACER -if (trace_hwlat_callback_enabled) -trace_hwlat_callback (true); -#endif +if (unlikely (trace_hwlat_callback_enabled)) +trace_hwlat_count_nmi (); } -static inline void ftrace_nmi_exit (void ) +static __always_inline void ftrace_nmi_handler_enter(void) { -#ifdef CONFIG_HWLAT_TRACER -if (trace_hwlat_callback_enabled) -trace_hwlat_callback (false); -#endif +if (unlikely (trace_hwlat_callback_enabled)) +trace_hwlat_timestamp (true); } +static __always_inline void ftrace_nmi_handler_exit(void) +{ +if (unlikely (trace_hwlat_callback_enabled)) +trace_hwlat_timestamp (false); +} +#else /* CONFIG_HWLAT_TRACER */ +static inline void ftrace_count_nmi (void ) {} +static inline void ftrace_nmi_handler_enter (void ) {} +static inline void ftrace_nmi_handler_exit (void ) {} +#endif +#endif /* _LINUX_FTRACE_IRQ_H */
include/linux/hardirq.h +3 -2
@@ -82,20 +82,21 @@ extern void irq_exit(void);arch_nmi_enter (); \printk_nmi_enter (); \lockdep_off (); \ -ftrace_nmi_enter (); \BUG_ON (in_nmi () ==NMI_MASK ); \__preempt_count_add (NMI_OFFSET +HARDIRQ_OFFSET ); \rcu_nmi_enter (); \lockdep_hardirq_enter (); \ +ftrace_count_nmi (); \ +ftrace_nmi_handler_enter (); \ } while (0)#define nmi_exit () \ do { \ + ftrace_nmi_handler_exit(); \ lockdep_hardirq_exit(); \ rcu_nmi_exit(); \ BUG_ON(!in_nmi()); \ __preempt_count_sub(NMI_OFFSET + HARDIRQ_OFFSET); \ - ftrace_nmi_exit(); \ lockdep_on(); \ printk_nmi_exit(); \ arch_nmi_exit(); \
kernel/trace/trace_hwlat.c +12 -7
@@ -132,21 +132,26 @@ static void trace_hwlat_sample(struct hw }/* +* Count NMIs in nmi_enter(). Does not take timestamps +* because the timestamping callchain cannot be invoked +* from noinstr sections. +*/ +noinstr void trace_hwlat_count_nmi(void) +{ +if (smp_processor_id () == nmi_cpu) + nmi_count++; +} + +/* * Timestamping uses ktime_get_mono_fast(), the NMI safe access to * CLOCK_MONOTONIC. */ -void trace_hwlat_callback (bool enter) +void trace_hwlat_timestamp (bool enter) { -if (smp_processor_id () != nmi_cpu) -return ; -if (enter) nmi_ts_start =ktime_get_mono_fast_ns ();else nmi_total_ts +=ktime_get_mono_fast_ns () - nmi_ts_start; - -if (enter) - nmi_count++; }/**
Re: [patch V6 02/37] tracing/hwlat: Split ftrace_nmi_enter/exit()
Quoted 11 lines · On Sat, 16 May 2020 01:45:49 +0200 Thomas Gleixner <[email protected]> wrote:
The hardware latency tracer calls into timekeeping and ends up in various instrumentable functions which is problematic vs. the kprobe handling especially the text poke machinery. It's invoked from nmi_enter/exit(), i.e. non-instrumentable code.
Split it into two parts:
1) NMI counter, only invoked on
nmi_enter()and noinstr safe2) NMI timestamping, to be invoked from instrumentable code
Quoted 8 lines
Move it into the rcu is watching regions of nmi_enter/exit() even if there is no actual RCU dependency right now but there is also no point in having it early.
The actual split of nmi_enter/exit() is done in a separate step.
Requested-by Steven Rostedt <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>
-- Steve
PATCH v6 3/37 nmi, tracing: Provide nmi_enter/exit_notrace()
In reply to Thomas GleixnerTo fully isolate #DB and #BP from instrumentable code it's necessary to avoid invoking the hardware latency tracer on nmi_enter/exit().
Provide nmi_enter/exit() variants which are not invoking the hardware latency tracer. That allows to put calls explicitely into the call sites outside of the kprobe handling.
--- include/linux/hardirq.h | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
include/linux/hardirq.h +21 -2
@@ -76,8 +76,16 @@ extern void irq_exit(void);/* * nmi_enter() can nest up to 15 times; see NMI_BITS. +* +* ftrace_count_nmi() only increments a counter and is noinstr safe so it +* can be invoked in nmi_enter_notrace(). ftrace_nmi_handler_enter/exit() +* does time stamping and will be invoked in the actual NMI handling after +* an instrumentable section has been reached. +* +* nmi_enter/exit() still calls into the tracer so existing callers +* wont break. */ -#define nmi_enter () \ +#define nmi_enter_notrace () \ do { \ arch_nmi_enter(); \ printk_nmi_enter(); \
@@ -87,10 +95,15 @@ extern void irq_exit(void);rcu_nmi_enter (); \lockdep_hardirq_enter (); \ftrace_count_nmi (); \ + } while (0) + +#define nmi_enter () \ + do { \ + nmi_enter_notrace(); \ ftrace_nmi_handler_enter(); \ } while (0) -#define nmi_exit () \ +#define nmi_exit_notrace () \ do { \ ftrace_nmi_handler_exit(); \ lockdep_hardirq_exit(); \
@@ -102,4 +115,10 @@ extern void irq_exit(void);arch_nmi_exit (); \ } while (0) +#define nmi_exit () \ + do { \ + ftrace_nmi_handler_exit(); \ + nmi_exit_notrace(); \ + } while (0) +#endif /* LINUX_HARDIRQ_H */
Re: [patch V6 03/37] nmi, tracing: Provide nmi_enter/exit_notrace()
Quoted 8 lines · Thomas Gleixner
To fully isolate #DB and #BP from instrumentable code it's necessary to avoid invoking the hardware latency tracer on nmi_enter/exit().
Provide nmi_enter/exit() variants which are not invoking the hardware latency tracer. That allows to put calls explicitely into the call sites outside of the kprobe handling.
Re: [patch V6 03/37] nmi, tracing: Provide nmi_enter/exit_notrace()
In reply to Thomas GleixnerQuoted 8 lines · On Sat, 16 May 2020 01:45:50 +0200 Thomas Gleixner <[email protected]> wrote:
To fully isolate #DB and #BP from instrumentable code it's necessary to avoid invoking the hardware latency tracer on nmi_enter/exit().
Provide nmi_enter/exit() variants which are not invoking the hardware latency tracer. That allows to put calls explicitely into the call sites outside of the kprobe handling.
Signed-off-by Thomas Gleixner <[email protected]>
-- Steve
PATCH v6 4/37 x86: Make hardware latency tracing explicit
In reply to Thomas GleixnerThe hardware latency tracer calls into trace_sched_clock and ends up in various instrumentable functions which is problemeatic vs. the kprobe handling especially the text poke machinery. It's invoked from nmi_enter/exit(), i.e. non-instrumentable code.
Use nmi_enter/exit_notrace() instead. These variants do not invoke the hardware latency tracer which avoids chasing down complex callchains to make them non-instrumentable.
The real interesting measurement is the actual NMI handler. Add an explicit invocation for the hardware latency tracer to it.
#DB and #BP are uninteresting as they really should not be in use when analzying hardware induced latencies.
If #DF hits, hardware latency is definitely not interesting anymore and in case of a machine check the hardware latency is not the most troublesome issue either.
--- arch/x86/kernel/cpu/mce/core.c | 4 ++-- arch/x86/kernel/nmi.c | 6 ++++-- arch/x86/kernel/traps.c | 12 +++++++----- 3 files changed, 13 insertions(+), 9 deletions(-)
arch/x86/kernel/cpu/mce/core.c +2 -2
@@ -1916,7 +1916,7 @@ static __always_inline void exc_machine_ mce_check_crashing_cpu())return ; -nmi_enter (); +nmi_enter_notrace ();/* * The call targets are marked noinstr, but objtool can't figure * that out because it's an indirect call. Annotate it.
@@ -1924,7 +1924,7 @@ static __always_inline void exc_machine_instrumentation_begin ();machine_check_vector (regs);instrumentation_end (); -nmi_exit (); +nmi_exit_notrace (); } static __always_inline void exc_machine_check_user(struct pt_regs *regs)
arch/x86/kernel/nmi.c +4 -2
@@ -334,6 +334,7 @@ static noinstr void default_do_nmi(struc__this_cpu_write (last_nmi_rip, regs->ip);instrumentation_begin (); +ftrace_nmi_handler_enter (); handled =nmi_handle (NMI_LOCAL , regs);__this_cpu_add (nmi_stats.normal, handled);
@@ -420,6 +421,7 @@ static noinstr void default_do_nmi(strucunknown_nmi_error (reason, regs); out: +ftrace_nmi_handler_exit ();instrumentation_end (); }
@@ -536,14 +538,14 @@ DEFINE_IDTENTRY_NMI(exc_nmi) }#endif -nmi_enter (); +nmi_enter_notrace ();inc_irq_stat (__nmi_count);if (!ignore_nmis)default_do_nmi (regs); -nmi_exit (); +nmi_exit_notrace ();#ifdef CONFIG_X86_64 if (unlikely (this_cpu_read (update_debug_stack))) {
arch/x86/kernel/traps.c +7 -5
@@ -387,7 +387,7 @@ DEFINE_IDTENTRY_DF(exc_double_fault) }#endif -nmi_enter (); +nmi_enter_notrace ();instrumentation_begin ();notify_die (DIE_TRAP , str, regs, error_code,X86_TRAP_DF ,SIGSEGV );
@@ -632,12 +632,14 @@ DEFINE_IDTENTRY_RAW(exc_int3)instrumentation_end ();idtentry_exit (regs); }else { -nmi_enter (); +nmi_enter_notrace ();instrumentation_begin (); +ftrace_nmi_handler_enter ();if (!do_int3 (regs))die ("int3" , regs,0 ); +ftrace_nmi_handler_exit ();instrumentation_end (); -nmi_exit (); +nmi_exit_notrace (); } }
@@ -849,7 +851,7 @@ static void noinstr handle_debug(struct static __always_inline void exc_debug_kernel(struct pt_regs *regs,unsigned long dr6) { -nmi_enter (); +nmi_enter_notrace ();/* * The SDM says "The processor clears the BTF flag when it * generates a debug exception." Clear TIF_BLOCKSTEP to keep
@@ -871,7 +873,7 @@ static __always_inline void exc_debug_keif (dr6)handle_debug (regs, dr6, false); -nmi_exit (); +nmi_exit_notrace (); } static __always_inline void exc_debug_user(struct pt_regs *regs,
Andy Lutomirski
maintainerRe: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 19 lines · Thomas Gleixner
The hardware latency tracer calls into
trace_sched_clockand ends up in various instrumentable functions which is problemeatic vs. the kprobe handling especially the text poke machinery. It's invoked from nmi_enter/exit(), i.e. non-instrumentable code.Use nmi_enter/exit_notrace() instead. These variants do not invoke the hardware latency tracer which avoids chasing down complex callchains to make them non-instrumentable.
The real interesting measurement is the actual NMI handler. Add an explicit invocation for the hardware latency tracer to it.
#DB and #BP are uninteresting as they really should not be in use when analzying hardware induced latencies.
arch/x86/kernel/traps.c static void noinstr handle_debug(struct · line 854
Quoted 5 lines
arch/x86/kernel/traps.c +1
-1@@ -849,7 +851,7 @@ static void noinstr handle_debug(struct static __always_inline void exc_debug_kernel(struct pt_regs *regs,unsigned long dr6) { -nmi_enter (); +nmi_enter_notrace ();
Why can't exc_debug_kernel() handle instrumentation? We shouldn't recurse into #DB since we've already cleared DR7, right?
Re: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 19 lines · Andy Lutomirski
Quoted 17 lines · Thomas Gleixner
The hardware latency tracer calls into
trace_sched_clockand ends up in various instrumentable functions which is problemeatic vs. the kprobe handling especially the text poke machinery. It's invoked from nmi_enter/exit(), i.e. non-instrumentable code.Use nmi_enter/exit_notrace() instead. These variants do not invoke the hardware latency tracer which avoids chasing down complex callchains to make them non-instrumentable.
The real interesting measurement is the actual NMI handler. Add an explicit invocation for the hardware latency tracer to it.
#DB and #BP are uninteresting as they really should not be in use when analzying hardware induced latencies.
Quoted 9 lines
arch/x86/kernel/traps.c static void noinstr handle_debug(struct · line 854
Quoted 6 lines
arch/x86/kernel/traps.c +1
-1@@ -849,7 +851,7 @@ static void noinstr handle_debug(struct static __always_inline void exc_debug_kernel(struct pt_regs *regs,unsigned long dr6) { -nmi_enter (); +nmi_enter_notrace ();Why can't
exc_debug_kernel()handle instrumentation? We shouldn't recurse into #DB since we've already cleared DR7, right?
It can later on. The point is that the trace stuff calls into the world and some more before the entry handling is complete.
Remember this is about ensuring that all the state is properly established before any of this instrumentation muck can happen.
DR7 handling is specific to #DB and done even before nmi_enter to prevent recursion.
Thanks,
tglx
Andy Lutomirski
maintainerRe: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 31 lines · Thomas Gleixner
Quoted 19 lines · Andy Lutomirski
Quoted 17 lines · Thomas Gleixner
The hardware latency tracer calls into
trace_sched_clockand ends up in various instrumentable functions which is problemeatic vs. the kprobe handling especially the text poke machinery. It's invoked from nmi_enter/exit(), i.e. non-instrumentable code.Use nmi_enter/exit_notrace() instead. These variants do not invoke the hardware latency tracer which avoids chasing down complex callchains to make them non-instrumentable.
The real interesting measurement is the actual NMI handler. Add an explicit invocation for the hardware latency tracer to it.
#DB and #BP are uninteresting as they really should not be in use when analzying hardware induced latencies.
Quoted 9 lines
arch/x86/kernel/traps.c static void noinstr handle_debug(struct · line 854
Quoted 6 lines
arch/x86/kernel/traps.c +1
-1@@ -849,7 +851,7 @@ static void noinstr handle_debug(struct static __always_inline void exc_debug_kernel(struct pt_regs *regs,unsigned long dr6) { -nmi_enter (); +nmi_enter_notrace ();Why can't
exc_debug_kernel()handle instrumentation? We shouldn't recurse into #DB since we've already cleared DR7, right?
Quoted 8 lines
It can later on. The point is that the trace stuff calls into the world and some more before the entry handling is complete.
Remember this is about ensuring that all the state is properly established before any of this instrumentation muck can happen.
DR7 handling is specific to #DB and done even before
nmi_enterto prevent recursion.
So why is this change needed?
Re: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 8 lines · Andy Lutomirski
Quoted 5 lines · Thomas Gleixner
Remember this is about ensuring that all the state is properly established before any of this instrumentation muck can happen.
DR7 handling is specific to #DB and done even before
nmi_enterto prevent recursion.So why is this change needed?
We really want nmi_enter() to be the carefully crafted mechanism which establishes correct state in whatever strange context the exception hits. Not more, not less.
Random instrumentation has absolutely no business there and I went a long way to make sure that this is enforcible by objtool.
Aside of that the tracing which is contained in nmi_enter() is about taking timestamps for hardware latency detection. If someone runs hardware latency detection with active break/watchpoints then I really can't help it.
Thanks,
tglx
Andy Lutomirski
maintainerRe: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 15 lines · Thomas Gleixner
Quoted 8 lines · Andy Lutomirski
Quoted 5 lines · Thomas Gleixner
Remember this is about ensuring that all the state is properly established before any of this instrumentation muck can happen.
DR7 handling is specific to #DB and done even before
nmi_enterto prevent recursion.So why is this change needed?
We really want
nmi_enter()to be the carefully crafted mechanism which establishes correct state in whatever strange context the exception hits. Not more, not less.
Quoted 8 lines
Random instrumentation has absolutely no business there and I went a long way to make sure that this is enforcible by objtool.
Aside of that the tracing which is contained in
nmi_enter()is about taking timestamps for hardware latency detection. If someone runs hardware latency detection with active break/watchpoints then I really can't help it.
Okay. I'll stop looking for the bug you're fixing, then.
Peter Zijlstra
maintainerRe: [patch V6 04/37] x86: Make hardware latency tracing explicit
In reply to Thomas GleixnerQuoted 11 lines · Thomas Gleixner
arch/x86/kernel/nmi.c +1
-0@@ -334,6 +334,7 @@ static noinstr void default_do_nmi(struc__this_cpu_write (last_nmi_rip, regs->ip);instrumentation_begin (); +ftrace_nmi_handler_enter (); handled =nmi_handle (NMI_LOCAL , regs);__this_cpu_add (nmi_stats.normal, handled);
arch/x86/kernel/nmi.c static noinstr void default_do_nmi(struc · line 426
Quoted 6 lines
arch/x86/kernel/nmi.c +1
-0@@ -420,6 +421,7 @@ static noinstr void default_do_nmi(strucunknown_nmi_error (reason, regs); out: +ftrace_nmi_handler_exit ();instrumentation_end (); }
Yeah, so I'm confused about this and the previous patch too. Why not do just this? Remove that ftrace_nmi_handler.* crud from nmi_{enter,exit}() and stick it here? Why do we needs the nmi_{enter,exit}_notrace() thing?
Re: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Thomas Gleixner
arch/x86/kernel/nmi.c +1
-0@@ -334,6 +334,7 @@ static noinstr void default_do_nmi(struc__this_cpu_write (last_nmi_rip, regs->ip);instrumentation_begin (); +ftrace_nmi_handler_enter (); handled =nmi_handle (NMI_LOCAL , regs);__this_cpu_add (nmi_stats.normal, handled);@@ -420,6 +421,7 @@ static noinstr void default_do_nmi(strucunknown_nmi_error (reason, regs);
Quoted 9 lines
arch/x86/kernel/nmi.c static noinstr void default_do_nmi(struc · line 426
Quoted 4 lines
arch/x86/kernel/nmi.c +1
-0out: +ftrace_nmi_handler_exit ();instrumentation_end (); }Yeah, so I'm confused about this and the previous patch too. Why not do just this? Remove that ftrace_nmi_handler.* crud from nmi_{enter,exit}() and stick it here? Why do we needs the nmi_{enter,exit}_notrace() thing?
Because you then have to fixup _all_ architectures which use nmi_enter/exit().
Thanks,
tglx
Peter Zijlstra
maintainerRe: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 25 lines · Thomas Gleixner
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Thomas Gleixner
arch/x86/kernel/nmi.c +1
-0@@ -334,6 +334,7 @@ static noinstr void default_do_nmi(struc__this_cpu_write (last_nmi_rip, regs->ip);instrumentation_begin (); +ftrace_nmi_handler_enter (); handled =nmi_handle (NMI_LOCAL , regs);__this_cpu_add (nmi_stats.normal, handled);@@ -420,6 +421,7 @@ static noinstr void default_do_nmi(strucunknown_nmi_error (reason, regs);Quoted 9 lines
arch/x86/kernel/nmi.c static noinstr void default_do_nmi(struc · line 426
Quoted 4 lines
arch/x86/kernel/nmi.c +1
-0out: +ftrace_nmi_handler_exit ();instrumentation_end (); }Yeah, so I'm confused about this and the previous patch too. Why not do just this? Remove that ftrace_nmi_handler.* crud from nmi_{enter,exit}() and stick it here? Why do we needs the nmi_{enter,exit}_notrace() thing?
Quoted 2 lines
Because you then have to fixup _all_ architectures which use nmi_enter/exit().
We probably have to anyway. But I can do that later I suppose.
Re: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Thomas Gleixner
Quoted 23 lines · Peter Zijlstra
Quoted 11 lines · Thomas Gleixner
arch/x86/kernel/nmi.c +1
-0@@ -334,6 +334,7 @@ static noinstr void default_do_nmi(struc__this_cpu_write (last_nmi_rip, regs->ip);instrumentation_begin (); +ftrace_nmi_handler_enter (); handled =nmi_handle (NMI_LOCAL , regs);__this_cpu_add (nmi_stats.normal, handled);arch/x86/kernel/nmi.c static noinstr void default_do_nmi(struc · line 426
Quoted 6 lines
arch/x86/kernel/nmi.c +1
-0@@ -420,6 +421,7 @@ static noinstr void default_do_nmi(strucunknown_nmi_error (reason, regs); out: +ftrace_nmi_handler_exit ();instrumentation_end (); }Yeah, so I'm confused about this and the previous patch too. Why not do just this? Remove that ftrace_nmi_handler.* crud from nmi_{enter,exit}() and stick it here? Why do we needs the nmi_{enter,exit}_notrace() thing?
Quoted 4 lines
Quoted 2 lines
Because you then have to fixup _all_ architectures which use nmi_enter/exit().
We probably have to anyway. But I can do that later I suppose.
Second thoughts. For #DB and #INT3 we can just keep nmi_enter(), needs just annotation in nmi_enter() around that trace muck.
For #NMI and #MCE I rather avoid the early trace call and do it once we have reached "stable" state, i.e. avoid it in the whole nested NMI mess.
Thanks,
tglx
Re: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 34 lines · Thomas Gleixner
Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Thomas Gleixner
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Thomas Gleixner
arch/x86/kernel/nmi.c +1
-0@@ -334,6 +334,7 @@ static noinstr void default_do_nmi(struc__this_cpu_write (last_nmi_rip, regs->ip);instrumentation_begin (); +ftrace_nmi_handler_enter (); handled =nmi_handle (NMI_LOCAL , regs);__this_cpu_add (nmi_stats.normal, handled);@@ -420,6 +421,7 @@ static noinstr void default_do_nmi(strucunknown_nmi_error (reason, regs);Quoted 9 lines
Quoted 4 lines
out:
+ ftrace_nmi_handler_exit(); instrumentation_end(); }Yeah, so I'm confused about this and the previous patch too. Why not do just this? Remove that ftrace_nmi_handler.* crud from nmi_{enter,exit}() and stick it here? Why do we needs the nmi_{enter,exit}_notrace() thing?
Because you then have to fixup _all_ architectures which use nmi_enter/exit().
Quoted 9 lines
Quoted 1 line
We probably have to anyway. But I can do that later I suppose.
Second thoughts. For #DB and #INT3 we can just keep
nmi_enter(), needs just annotation innmi_enter()around that trace muck.For #NMI and #MCE I rather avoid the early trace call and do it once we have reached "stable" state, i.e. avoid it in the whole nested NMI mess.
What’s the issue? The actual meat is mostly in the asm for NMI, and for MCE it’s just the sync-all-the-cores thing. The actual simultaneous NMI-and-MCE case is utterly busted regardless, and I’ve been thinking about how to fix it. It won’t be pretty, but nmi_enter() will have nothing to do with it.
Re: [patch V6 04/37] x86: Make hardware latency tracing explicit
Quoted 7 lines · Andy Lutomirski
Quoted 7 lines · Thomas Gleixner
Quoted 1 line · Peter Zijlstra
We probably have to anyway. But I can do that later I suppose.
Second thoughts. For #DB and #INT3 we can just keep
nmi_enter(), needs just annotation innmi_enter()around that trace muck.
Quoted 8 lines
Quoted 2 lines
For #NMI and #MCE I rather avoid the early trace call and do it once we have reached "stable" state, i.e. avoid it in the whole nested NMI mess.
What’s the issue? The actual meat is mostly in the asm for NMI, and for MCE it’s just the sync-all-the-cores thing. The actual simultaneous NMI-and-MCE case is utterly busted regardless, and I’ve been thinking about how to fix it. It won’t be pretty, but
nmi_enter()will have nothing to do with it.
The issue is that I want to avoid anything which is not essential just for pure paranoia reasons.
I can drop that and just move the trace muck after RCU is safe and annotate it properly.
PATCH v6 5/37 genirq: Provide irq_enter/exit_rcu()
In reply to Thomas Gleixnerirq_enter()/exit() include the RCU handling. To properly separate the RCU handling provide variants which contain only the non-RCU related functionality.
include/linux/hardirq.h +11 -2
@@ -43,7 +43,11 @@ extern void rcu_nmi_exit(void);/* * Enter irq context (on NO_HZ, update jiffies): */ -extern void irq_enter (void ); +void irq_enter (void ); +/* +* Like irq_enter(), but RCU is already watching. +*/ +void irq_enter_rcu (void );/* * Exit irq context without processing softirqs:
@@ -58,7 +62,12 @@ extern void irq_enter(void);/* * Exit irq context and process softirqs if needed: */ -extern void irq_exit (void ); +void irq_exit (void ); + +/* +* Like irq_exit(), but return with RCU watching. +*/ +void irq_exit_rcu (void );#ifndef arch_nmi_enter#define arch_nmi_enter () do { } while (0)
kernel/softirq.c +27 -8
@@ -339,12 +339,11 @@ asmlinkage __visible void do_softirq(void)local_irq_restore (flags); } -/* -* Enter an interrupt context. +/** +* irq_enter_rcu - Enter an interrupt context with RCU watching */ -void irq_enter (void ) +void irq_enter_rcu (void ) { -rcu_irq_enter ();if (is_idle_task (current) && !in_interrupt ()) {/* * Prevent raise_softirq from needlessly waking up ksoftirqd
@@ -354,10 +353,18 @@ void irq_enter(void)tick_irq_enter ();_local_bh_enable (); } -__irq_enter (); } +/** +* irq_enter - Enter an interrupt context including RCU update +*/ +void irq_enter (void ) +{ +rcu_irq_enter (); +irq_enter_rcu (); +} +static inline void invoke_softirq (void ) {if (ksoftirqd_running (local_softirq_pending ()))
@@ -397,10 +404,12 @@ static inline void tick_irq_exit(void)#endif } -/* -* Exit an interrupt context. Process softirqs if needed and possible: +/** +* irq_exit_rcu() - Exit an interrupt context without updating RCU +* +* Also processes softirqs if needed and possible. */ -void irq_exit (void ) +void irq_exit_rcu (void ) {#ifndef __ARCH_IRQ_EXIT_IRQS_DISABLEDlocal_irq_disable ();
@@ -413,6 +422,16 @@ void irq_exit(void)invoke_softirq ();tick_irq_exit (); +} + +/** +* irq_exit - Exit an interrupt context, update RCU and lockdep +* +* Also processes softirqs if needed and possible. +*/ +void irq_exit (void ) +{ +irq_exit_rcu ();rcu_irq_exit ();/* must be last! */ lockdep_hardirq_exit ();
Re: [patch V6 05/37] genirq: Provide irq_enter/exit_rcu()
Quoted 5 lines · Thomas Gleixner
irq_enter()/exit() include the RCU handling. To properly separate the RCU handling provide variants which contain only the non-RCU related functionality.
PATCH v6 6/37 genirq: Provde __irq_enter/exit_raw()
In reply to Thomas Gleixnerinclude/linux/hardirq.h +20 -0
@@ -41,6 +41,17 @@ extern void rcu_nmi_exit(void); }while (0 )/* +* Like __irq_enter() without time accounting for fast +* interrupts, e.g. reschedule IPI where time accounting +* is more expensive than the actual interrupt. + */ +#define __irq_enter_raw () \ + do { \ + preempt_count_add(HARDIRQ_OFFSET); \ + lockdep_hardirq_enter(); \ + } while (0) + +/* * Enter irq context (on NO_HZ, update jiffies): */ void irq_enter (void );
@@ -60,6 +71,15 @@ void irq_enter_rcu(void); }while (0 )/* +* Like __irq_exit() without time accounting + */ +#define __irq_exit_raw () \ + do { \ + lockdep_hardirq_exit(); \ + preempt_count_sub(HARDIRQ_OFFSET); \ + } while (0) + +/* * Exit irq context and process softirqs if needed: */ void irq_exit (void );
Re: [patch V6 06/37] genirq: Provde __irq_enter/exit_raw()
Quoted 1 line · Thomas Gleixner
PATCH v6 7/37 x86/entry: Provide helpers for execute on irqstack
In reply to Thomas GleixnerDevice interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.
arch/x86/entry/entry_64.S +39 -0
@@ -1106,6 +1106,45 @@ SYM_CODE_START_LOCAL_NOALIGN(.Lbad_gs) SYM_CODE_END(.Lbad_gs) .previous +/* + * rdi: New stack pointer points to the top word of the stack + * rsi: Function pointer + * rdx: Function argument (can be NULL if none) + */ +SYM_FUNC_START(asm_call_on_stack) + /* + * Save the frame pointer unconditionally. This allows the ORC + * unwinder to handle the stack switch. + */ + pushq %rbp + mov %rsp, %rbp + + /* + * The unwinder relies on the word at the top of the new stack + * page linking back to the previous RSP. + */ + mov %rsp, (%rdi) + mov %rdi, %rsp + /* Move the argument to the right place */ + mov %rdx, %rdi + +1: + .pushsection .discard.instr_begin + .long 1b - . + .popsection + + CALL_NOSPEC rsi + +2: + .pushsection .discard.instr_end + .long 2b - . + .popsection + + /* Restore the previous stack pointer from RBP. */ + leaveq + ret +SYM_FUNC_END(asm_call_on_stack) + /* Call softirq on interrupt stack. Interrupts are off. */ .pushsection .text, "ax" SYM_FUNC_START(do_softirq_own_stack)
arch/x86/include/asm/irq_stack.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_IRQ_STACK_H +#define _ASM_X86_IRQ_STACK_H + +#include <linux/ptrace.h> + +#include <asm/processor.h> + +#ifdef CONFIG_X86_64 +static __always_inline bool irqstack_active(void) +{ +return __this_cpu_read (irq_count) !=-1 ; +} + +void asm_call_on_stack (void *sp,void *func,void *arg); + +static __always_inline void run_on_irqstack(void *func, void *arg) +{ +void *tos =__this_cpu_read (hardirq_stack_ptr); + +lockdep_assert_irqs_disabled (); + +__this_cpu_add (irq_count,1 ); +asm_call_on_stack (tos -8 , func, arg); +__this_cpu_sub (irq_count,1 ); +} + +#else /* CONFIG_X86_64 */ +static inline bool irqstack_active (void ) {return false; } +static inline void run_on_irqstack (void *func,void *arg) { } +#endif /* !CONFIG_X86_64 */ + +static __always_inline bool irq_needs_irq_stack(struct pt_regs *regs) +{ +if (IS_ENABLED (CONFIG_X86_32 )) +return false; +return !user_mode (regs) && !irqstack_active (); +} + +#endif
Re: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
Quoted 23 lines · Thomas Gleixner
Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Quoted 7 lines
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.
Have you tested by forcing a stack trace from the IRQ stack and making sure it unwinds all the way out?
Andy Lutomirski
maintainerRe: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
Quoted 33 lines · Andy Lutomirski
Quoted 23 lines · Thomas Gleixner
Device interrupt handlers and system vector handlers are executed on the interrupt stack. The stack switch happens in the low level assembly entry code. This conflicts with the efforts to consolidate the exit code in C to ensure correctness vs. RCU and tracing.
As there is no way to move #DB away from IST due to the MOV SS issue, the requirements vs. #DB and NMI for switching to the interrupt stack do not exist anymore. The only requirement is that interrupts are disabled.
That allows to move the stack switching to C code which simplifies the entry/exit handling further because it allows to switch stacks after handling the entry and on exit before handling RCU, return to usermode and kernel preemption in the same way as for regular exceptions.
The initial attempt of having the stack switching in inline ASM caused too much headache vs. objtool and the unwinder. After analysing the use cases it was agreed on that having the stack switch in ASM for the price of an indirect call is acceptable as the main users are indirect call heavy anyway and the few system vectors which are empty shells (scheduler IPI and KVM posted interrupt vectors) can run from the regular stack.
Quoted 7 lines
Provide helper functions to check whether the interrupt stack is already active and whether stack switching is required.
64 bit only for now. 32 bit has a variant of that already. Once this is cleaned up the two implementations might be consolidated as a cleanup on top.
Quoted 4 lines
Acked-by Andy Lutomirski <[email protected]>Have you tested by forcing a stack trace from the IRQ stack and making sure it unwinds all the way out?
Actually, I revoke my ack. Can you make one of two changes:
Option A: Add an assertion to run_on_irqstack to verify that irq_count was -1 at the beginning? I suppose this also means you could just explicitly write 0 instead of adding and subtracting.
Option B: Make run_on_irqstack() just call the function on the current stack if we're already on the irq stack.
Right now, it's too easy to mess up and not verify the right precondition before calling run_on_irqstack().
If you choose A, perhaps add a helper to do the if(irq_needs_irqstack) dance so that users can just do:
run_on_irqstack_if_needed(...);
instead of checking everything themselves.
Re: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
Quoted 12 lines · Andy Lutomirski
Actually, I revoke my ack. Can you make one of two changes:
Option A: Add an assertion to
run_on_irqstackto verify thatirq_countwas -1 at the beginning? I suppose this also means you could just explicitly write 0 instead of adding and subtracting.Option B: Make
run_on_irqstack()just call the function on the current stack if we're already on the irq stack.Right now, it's too easy to mess up and not verify the right precondition before calling
run_on_irqstack().
Quoted 6 lines
If you choose A, perhaps add a helper to do the
if(irq_needs_irqstack)dance so that users can just do:
run_on_irqstack_if_needed(...);instead of checking everything themselves.
I'll have a look tomorrow morning with brain awake.
Andy Lutomirski
maintainerRe: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Actually, I revoke my ack. Can you make one of two changes:
Option A: Add an assertion to
run_on_irqstackto verify thatirq_countwas -1 at the beginning? I suppose this also means you could just explicitly write 0 instead of adding and subtracting.Option B: Make
run_on_irqstack()just call the function on the current stack if we're already on the irq stack.Right now, it's too easy to mess up and not verify the right precondition before calling
run_on_irqstack().
Quoted 8 lines
Quoted 6 lines
If you choose A, perhaps add a helper to do the
if(irq_needs_irqstack)dance so that users can just do:
run_on_irqstack_if_needed(...);instead of checking everything themselves.
I'll have a look tomorrow morning with brain awake.
Also, reading more of the series, I suspect that asm_call_on_stack is logically in the wrong section or that the noinstr stuff is otherwise not quite right. I think that objtool should not accept run_on_irqstack() from noinstr code. See followups on patch 10.
Re: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
Quoted 20 lines · Andy Lutomirski
Quoted 19 lines · Thomas Gleixner
Quoted 17 lines · Andy Lutomirski
Actually, I revoke my ack. Can you make one of two changes:
Option A: Add an assertion to
run_on_irqstackto verify thatirq_countwas -1 at the beginning? I suppose this also means you could just explicitly write 0 instead of adding and subtracting.Option B: Make
run_on_irqstack()just call the function on the current stack if we're already on the irq stack.Right now, it's too easy to mess up and not verify the right precondition before calling
run_on_irqstack().If you choose A, perhaps add a helper to do the
if(irq_needs_irqstack)dance so that users can just do:
run_on_irqstack_if_needed(...);
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
instead of checking everything themselves.
I'll have a look tomorrow morning with brain awake.
Also, reading more of the series, I suspect that
asm_call_on_stackis logically in the wrong section or that the noinstr stuff is otherwise not quite right. I think that objtool should not acceptrun_on_irqstack()from noinstr code. See followups on patch 10.
It's in entry.text which is non-instrumentable as well.
Andy Lutomirski
maintainerRe: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
Quoted 24 lines · Thomas Gleixner
Quoted 22 lines · Andy Lutomirski
Quoted 21 lines · Thomas Gleixner
Quoted 18 lines · Andy Lutomirski
Actually, I revoke my ack. Can you make one of two changes:
Option A: Add an assertion to
run_on_irqstackto verify thatirq_countwas -1 at the beginning? I suppose this also means you could just explicitly write 0 instead of adding and subtracting.Option B: Make
run_on_irqstack()just call the function on the current stack if we're already on the irq stack.Right now, it's too easy to mess up and not verify the right precondition before calling
run_on_irqstack().If you choose A, perhaps add a helper to do the
if(irq_needs_irqstack)dance so that users can just do:
run_on_irqstack_if_needed(...);instead of checking everything themselves.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
I'll have a look tomorrow morning with brain awake.
Also, reading more of the series, I suspect that
asm_call_on_stackis logically in the wrong section or that the noinstr stuff is otherwise not quite right. I think that objtool should not acceptrun_on_irqstack()from noinstr code. See followups on patch 10.It's in entry.text which is non-instrumentable as well.
Hmm. I suppose we can chalk this up to the noinstr checking not being entirely perfect.
Re: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
Quoted 32 lines · Andy Lutomirski
Quoted 31 lines · Thomas Gleixner
Quoted 28 lines · Andy Lutomirski
Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Actually, I revoke my ack. Can you make one of two changes:
Option A: Add an assertion to
run_on_irqstackto verify thatirq_countwas -1 at the beginning? I suppose this also means you could just explicitly write 0 instead of adding and subtracting.Option B: Make
run_on_irqstack()just call the function on the current stack if we're already on the irq stack.Right now, it's too easy to mess up and not verify the right precondition before calling
run_on_irqstack().Quoted 8 lines
Quoted 6 lines
If you choose A, perhaps add a helper to do the
if(irq_needs_irqstack)dance so that users can just do:
run_on_irqstack_if_needed(...);instead of checking everything themselves.
I'll have a look tomorrow morning with brain awake.
Also, reading more of the series, I suspect that
asm_call_on_stackis logically in the wrong section or that the noinstr stuff is otherwise not quite right. I think that objtool should not acceptrun_on_irqstack()from noinstr code. See followups on patch 10.
Quoted 4 lines
Quoted 1 line
It's in entry.text which is non-instrumentable as well.
Hmm. I suppose we can chalk this up to the noinstr checking not being entirely perfect.
objtool considers both entry.text and noinstr.text. We just can't stick everything into entry.text for these !%@#45@# reasons.
Andy Lutomirski
maintainerRe: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
Quoted 34 lines · Thomas Gleixner
Quoted 32 lines · Andy Lutomirski
Quoted 31 lines · Thomas Gleixner
Quoted 28 lines · Andy Lutomirski
Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Actually, I revoke my ack. Can you make one of two changes:
Option A: Add an assertion to
run_on_irqstackto verify thatirq_countwas -1 at the beginning? I suppose this also means you could just explicitly write 0 instead of adding and subtracting.Option B: Make
run_on_irqstack()just call the function on the current stack if we're already on the irq stack.Right now, it's too easy to mess up and not verify the right precondition before calling
run_on_irqstack().Quoted 8 lines
Quoted 6 lines
If you choose A, perhaps add a helper to do the
if(irq_needs_irqstack)dance so that users can just do:
run_on_irqstack_if_needed(...);instead of checking everything themselves.
I'll have a look tomorrow morning with brain awake.
Also, reading more of the series, I suspect that
asm_call_on_stackis logically in the wrong section or that the noinstr stuff is otherwise not quite right. I think that objtool should not acceptrun_on_irqstack()from noinstr code. See followups on patch 10.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
It's in entry.text which is non-instrumentable as well.
Hmm. I suppose we can chalk this up to the noinstr checking not being entirely perfect.
objtool considers both entry.text and noinstr.text. We just can't stick everything into entry.text for these !%@#45@# reasons.
Meh, this is all fine I think. I think it would be slightly nicer if objtool were to warn if noinstr code called run_on_stack(), but I'm not sure it matters much.
Re: [patch V6 07/37] x86/entry: Provide helpers for execute on irqstack
In reply to Andy LutomirskiQuoted 4 lines · Andy Lutomirski
On Fri, May 15, 2020 at 5:10 PM Thomas Gleixner <[email protected]> wrote:
Have you tested by forcing a stack trace from the IRQ stack and making sure it unwinds all the way out?
Yes.
PATCH v6 8/37 x86/entry/64: Move do_softirq_own_stack() to C
In reply to Thomas GleixnerThe first step to get rid of the ENTER/LEAVE_IRQ_STACK ASM macro maze. Use the new C code helpers to move do_softirq_own_stack() out of ASM code.
arch/x86/entry/entry_64.S +0 -13
@@ -1145,19 +1145,6 @@ SYM_FUNC_START(asm_call_on_stack) ret SYM_FUNC_END(asm_call_on_stack) -/* Call softirq on interrupt stack. Interrupts are off. */ -.pushsection .text, "ax" -SYM_FUNC_START(do_softirq_own_stack) - pushq %rbp - mov %rsp, %rbp - ENTER_IRQ_STACK regs=0 old_rsp=%r11 - call __do_softirq - LEAVE_IRQ_STACK regs=0 - leaveq - ret -SYM_FUNC_END(do_softirq_own_stack) -.popsection - #ifdef CONFIG_XEN_PV /* * A note on the "critical region" in our callback handler.
arch/x86/kernel/irq_64.c +9 -0
@@ -20,6 +20,7 @@#include <linux/sched/task_stack.h> #include <asm/cpu_entry_area.h> +#include <asm/irq_stack.h> #include <asm/io_apic.h> #include <asm/apic.h>
@@ -70,3 +71,11 @@ int irq_init_percpu_irqstack(unsigned int cpu)return 0 ;return map_irq_stack (cpu); } + +void do_softirq_own_stack (void ) +{ +if (irqstack_active ()) +__do_softirq (); +else +run_on_irqstack (__do_softirq,NULL ); +}
Andy Lutomirski
maintainerRe: [patch V6 08/37] x86/entry/64: Move do_softirq_own_stack() to C
Quoted 45 lines · Thomas Gleixner
The first step to get rid of the ENTER/LEAVE_IRQ_STACK ASM macro maze. Use the new C code helpers to move
do_softirq_own_stack()out of ASM code.Signed-off-by Thomas Gleixner <[email protected]>arch/x86/entry/entry_64.S +0
-13@@ -1145,19 +1145,6 @@ SYM_FUNC_START(asm_call_on_stack) ret SYM_FUNC_END(asm_call_on_stack) -/* Call softirq on interrupt stack. Interrupts are off. */ -.pushsection .text, "ax" -SYM_FUNC_START(do_softirq_own_stack) - pushq %rbp - mov %rsp, %rbp - ENTER_IRQ_STACK regs=0 old_rsp=%r11 - call __do_softirq - LEAVE_IRQ_STACK regs=0 - leaveq - ret -SYM_FUNC_END(do_softirq_own_stack) -.popsection - #ifdef CONFIG_XEN_PV /* * A note on the "critical region" in our callback handler.arch/x86/kernel/irq_64.c +1
-0@@ -20,6 +20,7 @@#include <linux/sched/task_stack.h> #include <asm/cpu_entry_area.h> +#include <asm/irq_stack.h> #include <asm/io_apic.h> #include <asm/apic.h> @@ -70,3 +71,11 @@ int irq_init_percpu_irqstack(unsigned int cpu)return 0 ;
arch/x86/kernel/irq_64.c int irq_init_percpu_irqstack(unsigned int cpu) · line 81
Quoted 10 lines
arch/x86/kernel/irq_64.c +8
-0return map_irq_stack (cpu); } + +void do_softirq_own_stack (void ) +{ +if (irqstack_active ()) +__do_softirq (); +else +run_on_irqstack (__do_softirq,NULL ); +}
See my comment in patch 8. I see no great reason that this should open-code the conditional, except maybe that we don't have pt_regs here so the condition needs to be a bit different.
PATCH v6 9/37 x86/entry: Split idtentry_enter/exit()
In reply to Thomas GleixnerSplit the implementation of idtentry_enter/exit() out into inline functions so that variants of idtentry_enter/exit() can be implemented without duplicating code.
arch/x86/entry/common.c +21 -16
@@ -539,22 +539,7 @@ void noinstr idtentry_enter(struct pt_regs *regs) } } -/** -* idtentry_exit - Common code to handle return from exceptions -* @regs: Pointer to pt_regs (exception entry regs) -* -* Depending on the return target (kernel/user) this runs the necessary -* preemption and work checks if possible and required and returns to -* the caller with interrupts disabled and no further work pending. -* -* This is the last action before returning to the low level ASM code which -* just needs to return to the appropriate context. -* -* Invoked by all exception/interrupt IDTENTRY handlers which are not -* returning through the paranoid exit path (all except NMI, #DF and the IST -* variants of #MC and #DB) and are therefore on the thread stack. -*/ -void noinstr idtentry_exit(struct pt_regs *regs) +static __always_inline void __idtentry_exit(struct pt_regs *regs) {lockdep_assert_irqs_disabled ();
@@ -609,3 +594,23 @@ void noinstr idtentry_exit(struct pt_regs *regs)rcu_irq_exit (); } } + +/** +* idtentry_exit - Common code to handle return from exceptions +* @regs: Pointer to pt_regs (exception entry regs) +* +* Depending on the return target (kernel/user) this runs the necessary +* preemption and work checks if possible and required and returns to +* the caller with interrupts disabled and no further work pending. +* +* This is the last action before returning to the low level ASM code which +* just needs to return to the appropriate context. +* +* Invoked by all exception/interrupt IDTENTRY handlers which are not +* returning through the paranoid exit path (all except NMI, #DF and the IST +* variants of #MC and #DB) and are therefore on the thread stack. +*/ +void noinstr idtentry_exit(struct pt_regs *regs) +{ +__idtentry_exit (regs); +}
Andy Lutomirski
maintainerRe: [patch V6 09/37] x86/entry: Split idtentry_enter/exit()
Quoted 6 lines · Thomas Gleixner
Split the implementation of idtentry_enter/exit() out into inline functions so that variants of idtentry_enter/exit() can be implemented without duplicating code.
After reading just this patch, I don't see how it helps anything. Maybe it'll make more sense after I read more of the series.
Re: [patch V6 09/37] x86/entry: Split idtentry_enter/exit()
Quoted 9 lines · Andy Lutomirski
Quoted 6 lines · Thomas Gleixner
Split the implementation of idtentry_enter/exit() out into inline functions so that variants of idtentry_enter/exit() can be implemented without duplicating code.
After reading just this patch, I don't see how it helps anything. Maybe it'll make more sense after I read more of the series.
I hope so :)
PATCH v6 10/37 x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
In reply to Thomas GleixnerConvert the XEN/PV hypercall to IDTENTRY:
- Emit the ASM stub with DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Remove the old prototypes
The handler stubs need to stay in ASM code as it needs corner case handling and adjustment of the stack pointer.
Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack.
The exit code is slightly different from the regular idtentry_exit() on non-preemptible kernels. If the hypercall is preemptible and need_resched() is set then XEN provides a preempt hypercall scheduling function. Add it as conditional path to __idtentry_exit() so the function can be reused.
__idtentry_exit() is forced inlined so on the regular idtentry_exit() path the extra condition is optimized out by the compiler.
arch/x86/entry/common.c +55 -2
@@ -27,6 +27,9 @@#include <linux/syscalls.h> #include <linux/uaccess.h> +#include <xen/xen-ops.h> +#include <xen/events.h> +#include <asm/desc.h> #include <asm/traps.h> #include <asm/vdso.h>
@@ -35,6 +38,7 @@#include <asm/nospec-branch.h> #include <asm/io_bitmap.h> #include <asm/syscall.h> +#include <asm/irq_stack.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
@@ -539,7 +543,8 @@ void noinstr idtentry_enter(struct pt_regs *regs) } } -static __always_inline void __idtentry_exit(struct pt_regs *regs) +static __always_inline void __idtentry_exit(struct pt_regs *regs, +bool preempt_hcall) {lockdep_assert_irqs_disabled ();
@@ -573,6 +578,16 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs)instrumentation_end ();return ; } + }else if (IS_ENABLED (CONFIG_XEN_PV )) { +if (preempt_hcall) { +/* See CONFIG_PREEMPTION above */ +instrumentation_begin (); +rcu_irq_exit_preempt (); +xen_maybe_preempt_hcall (); +trace_hardirqs_on (); +instrumentation_end (); +return ; + } }/* * If preemption is disabled then this needs to be done
@@ -612,5 +627,43 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs)*/ void noinstr idtentry_exit(struct pt_regs *regs) { -__idtentry_exit (regs); +__idtentry_exit (regs, false); +} + +#ifdef CONFIG_XEN_PV +static void __xen_pv_evtchn_do_upcall (void ) +{ +irq_enter_rcu (); +inc_irq_stat (irq_hv_callback_count); + +xen_hvm_evtchn_do_upcall (); + +irq_exit_rcu (); +} + +__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; + +idtentry_enter (regs); + old_regs =set_irq_regs (regs); + +if (!irq_needs_irq_stack (regs)) { +instrumentation_begin (); +__xen_pv_evtchn_do_upcall (); +instrumentation_end (); + }else { +run_on_irqstack (__xen_pv_evtchn_do_upcall,NULL ); + } + +set_irq_regs (old_regs); + +if (IS_ENABLED (CONFIG_PREEMPTION )) { +__idtentry_exit (regs, false); + }else { +bool inhcall =__this_cpu_read (xen_in_preemptible_hcall); + +__idtentry_exit (regs, inhcall &&need_resched ()); + } } +#endif /* CONFIG_XEN_PV */
arch/x86/entry/entry_32.S +9 -8
@@ -1298,7 +1298,10 @@ SYM_CODE_END(native_iret) #endif #ifdef CONFIG_XEN_PV -SYM_FUNC_START(xen_hypervisor_callback) +/* + * See comment in entry_64.S for further explanation + */ +SYM_FUNC_START(exc_xen_hypervisor_callback) /* * Check to see if we got the event in the critical * region in xen_iret_direct, after we've reenabled
@@ -1315,14 +1318,11 @@ SYM_FUNC_START(xen_hypervisor_callback) pushl $-1 /* orig_ax = -1 => not a system call */ SAVE_ALL ENCODE_FRAME_POINTER - TRACE_IRQS_OFF + mov %esp, %eax - call xen_evtchn_do_upcall -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp ret_from_intr -SYM_FUNC_END(xen_hypervisor_callback) + call xen_pv_evtchn_do_upcall + jmp handle_exception_return +SYM_FUNC_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
@@ -1464,6 +1464,7 @@ SYM_CODE_START_LOCAL_NOALIGN(handle_exception) movl %esp, %eax # pt_regs pointer CALL_NOSPEC edi +handle_exception_return: #ifdef CONFIG_VM86 movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS movb PT_CS(%esp), %al
arch/x86/entry/entry_64.S +6 -14
@@ -1067,10 +1067,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1 -#ifdef CONFIG_XEN_PV -idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0 -#endif - /* * Reload gs selector with exception handling * edi: new selector
@@ -1158,9 +1154,10 @@ SYM_FUNC_END(asm_call_on_stack) * So, on entry to the handler we detect whether we interrupted an * existing activation in its critical region -- if so, we pop the current * activation and restart the handler using the previous one. + * + * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs) */ -/* do_hypervisor_callback(struct *pt_regs) */ -SYM_CODE_START_LOCAL(xen_do_hypervisor_callback) +SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) /* * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1170,15 +1167,10 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_callback) movq %rdi, %rsp /* we don't return, adjust the stack frame */ UNWIND_HINT_REGS - ENTER_IRQ_STACK old_rsp=%r10 - call xen_evtchn_do_upcall - LEAVE_IRQ_STACK + call xen_pv_evtchn_do_upcall -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp error_exit -SYM_CODE_END(xen_do_hypervisor_callback) + jmp error_return +SYM_CODE_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
arch/x86/include/asm/idtentry.h +13 -0
@@ -332,6 +332,13 @@ __visible noinstr void func(struct pt_regs *regs, \* This avoids duplicate defines and ensures that everything is consistent. */ +/* +* Dummy trap number so the low level ASM macro vector number checks do not +* match which results in emitting plain IDTENTRY stubs without bells and +* whistels. +*/ +#define X86_TRAP_OTHER 0xFFFF +/* Simple exception entry points. No hardware error code */ DECLARE_IDTENTRY (X86_TRAP_DE , exc_divide_error);DECLARE_IDTENTRY (X86_TRAP_OF , exc_overflow);
@@ -373,4 +380,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB, debug);DECLARE_IDTENTRY_DF (X86_TRAP_DF , exc_double_fault);#endif +#ifdef CONFIG_XEN_PV +DECLARE_IDTENTRY (X86_TRAP_OTHER , exc_xen_hypervisor_callback); +#endif + +#undef X86_TRAP_OTHER +#endif
arch/x86/xen/setup.c +3 -1
@@ -20,6 +20,7 @@#include <asm/setup.h> #include <asm/acpi.h> #include <asm/numa.h> +#include <asm/idtentry.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h>
@@ -993,7 +994,8 @@ static void __init xen_pvmmu_arch_setup(void)HYPERVISOR_vm_assist (VMASST_CMD_enable, VMASST_TYPE_pae_extended_cr3); -if (register_callback (CALLBACKTYPE_event, xen_hypervisor_callback) || +if (register_callback (CALLBACKTYPE_event, + xen_asm_exc_xen_hypervisor_callback) ||register_callback (CALLBACKTYPE_failsafe, xen_failsafe_callback))BUG ();
arch/x86/xen/smp_pv.c +2 -1
@@ -27,6 +27,7 @@#include <asm/paravirt.h> #include <asm/desc.h> #include <asm/pgtable.h> +#include <asm/idtentry.h> #include <asm/cpu.h> #include <xen/interface/xen.h>
@@ -347,7 +348,7 @@ cpu_initialize_context(unsigned int cpu, struct task_struct *idle) ctxt->gs_base_kernel =per_cpu_offset (cpu);#endif ctxt->event_callback_eip = - (unsigned long )xen_hypervisor_callback; + (unsigned long )xen_asm_exc_xen_hypervisor_callback; ctxt->failsafe_callback_eip = (unsigned long )xen_failsafe_callback;per_cpu (xen_cr3, cpu) =__pa (swapper_pg_dir);
arch/x86/xen/xen-asm_32.S +7 -5
@@ -93,7 +93,7 @@ xen_iret_start_crit: /* * If there's something pending, mask events again so we can - * jump back into xen_hypervisor_callback. Otherwise do not + * jump back into exc_xen_hypervisor_callback. Otherwise do not * touch XEN_vcpu_info_mask. */ jne 1f
@@ -113,7 +113,7 @@ iret_restore_end: * Events are masked, so jumping out of the critical region is * OK. */ - je xen_hypervisor_callback + je asm_exc_xen_hypervisor_callback 1: iret xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret) .globl xen_iret_start_crit, xen_iret_end_crit /* - * This is called by xen_hypervisor_callback in entry_32.S when it sees + * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees * that the EIP at the time of interrupt was between * xen_iret_start_crit and xen_iret_end_crit. *
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret) * eflags } * cs } nested exception info * eip } - * return address : (into xen_hypervisor_callback) + * return address : (into asm_exc_xen_hypervisor_callback) * * In order to deliver the nested exception properly, we need to discard the * nested exception frame such that when we handle the exception, we do it
@@ -152,7 +152,8 @@ SYM_CODE_END(xen_iret) * * The only caveat is that if the outer eax hasn't been restored yet (i.e. * it's still on stack), we need to restore its value here. - */ +*/ +.pushsection .noinstr.text, "ax" SYM_CODE_START(xen_iret_crit_fixup) /* * Paranoia: Make sure we're really coming from kernel space.
@@ -181,3 +182,4 @@ SYM_CODE_START(xen_iret_crit_fixup) 2: ret SYM_CODE_END(xen_iret_crit_fixup) +.popsection
arch/x86/xen/xen-asm_64.S +1 -1
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_error #ifdef CONFIG_IA32_EMULATION xen_pv_trap entry_INT80_compat #endif -xen_pv_trap hypervisor_callback +xen_pv_trap asm_exc_xen_hypervisor_callback __INIT SYM_CODE_START(xen_early_idt_handler_array)
arch/x86/xen/xen-ops.h +0 -1
@@ -8,7 +8,6 @@#include <xen/xen-ops.h> /* These are code, but not functions. Defined in entry.S */ -extern const char xen_hypervisor_callback[];extern const char xen_failsafe_callback[];void xen_sysenter_target (void );
drivers/xen/preempt.c +1 -1
@@ -24,7 +24,7 @@DEFINE_PER_CPU (bool, xen_in_preemptible_hcall);EXPORT_SYMBOL_GPL (xen_in_preemptible_hcall); -asmlinkage __visible void xen_maybe_preempt_hcall(void) +void xen_maybe_preempt_hcall (void ) {if (unlikely (__this_cpu_read (xen_in_preemptible_hcall) &&need_resched ())) {
include/xen/xen-ops.h +5 -2
@@ -214,6 +214,7 @@ bool xen_running_on_version_or_later(unsigned int major, unsigned int minor);void xen_efi_runtime_setup (void ); +DECLARE_PER_CPU (bool, xen_in_preemptible_hcall);#ifdef CONFIG_PREEMPTION
@@ -225,9 +226,9 @@ static inline void xen_preemptible_hcall_end(void) { } -#else +static inline void xen_maybe_preempt_hcall (void ) { } -DECLARE_PER_CPU (bool, xen_in_preemptible_hcall); +#else static inline void xen_preemptible_hcall_begin (void ) {
@@ -239,6 +240,8 @@ static inline void xen_preemptible_hcall_end(void)__this_cpu_write (xen_in_preemptible_hcall, false); } +void xen_maybe_preempt_hcall (void ); +#endif /* CONFIG_PREEMPTION */ #endif /* INCLUDE_XEN_OPS_H */
Andy Lutomirski
maintainerRe: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 63 lines · Thomas Gleixner
Convert the XEN/PV hypercall to IDTENTRY:
- Emit the ASM stub with
DECLARE_IDTENTRY
- Remove the ASM idtentry in 64bit
- Remove the open coded ASM entry code in 32bit
- Remove the old prototypesThe handler stubs need to stay in ASM code as it needs corner case handling and adjustment of the stack pointer.
Provide a new C function which invokes the entry/exit handling and calls into the XEN handler on the interrupt stack.
The exit code is slightly different from the regular
idtentry_exit()on non-preemptible kernels. If the hypercall is preemptible andneed_resched()is set then XEN provides a preempt hypercall scheduling function. Add it as conditional path to__idtentry_exit()so the function can be reused.
__idtentry_exit()is forced inlined so on the regularidtentry_exit()path the extra condition is optimized out by the compiler.Signed-off-by Thomas Gleixner <[email protected]>Cc Boris Ostrovsky <[email protected]>Cc Juergen Gross <[email protected]>arch/x86/entry/common.c +6
-1@@ -27,6 +27,9 @@#include <linux/syscalls.h> #include <linux/uaccess.h> +#include <xen/xen-ops.h> +#include <xen/events.h> +#include <asm/desc.h> #include <asm/traps.h> #include <asm/vdso.h> @@ -35,6 +38,7 @@#include <asm/nospec-branch.h> #include <asm/io_bitmap.h> #include <asm/syscall.h> +#include <asm/irq_stack.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h> @@ -539,7 +543,8 @@ void noinstr idtentry_enter(struct pt_regs *regs) } } -static __always_inline void __idtentry_exit(struct pt_regs *regs) +static __always_inline void __idtentry_exit(struct pt_regs *regs, +bool preempt_hcall) {lockdep_assert_irqs_disabled ();@@ -573,6 +578,16 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs)instrumentation_end ();return ; }
arch/x86/entry/common.c static __always_inline void __idtentry_exit(struct pt_regs *regs) · line 590
Quoted 10 lines
arch/x86/entry/common.c +10
-0+ }else if (IS_ENABLED (CONFIG_XEN_PV )) { +if (preempt_hcall) { +/* See CONFIG_PREEMPTION above */ +instrumentation_begin (); +rcu_irq_exit_preempt (); +xen_maybe_preempt_hcall (); +trace_hardirqs_on (); +instrumentation_end (); +return ; + }
Ewwwww! This shouldn't be taken as a NAK -- it's just an expression of disgust.
Quoted 26 lines
arch/x86/entry/common.c +18
-1}/* * If preemption is disabled then this needs to be done @@ -612,5 +627,43 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs)*/ void noinstr idtentry_exit(struct pt_regs *regs) { -__idtentry_exit (regs); +__idtentry_exit (regs, false); +} + +#ifdef CONFIG_XEN_PV +static void __xen_pv_evtchn_do_upcall (void ) +{ +irq_enter_rcu (); +inc_irq_stat (irq_hv_callback_count); + +xen_hvm_evtchn_do_upcall (); + +irq_exit_rcu (); +} + +__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +
arch/x86/entry/common.c static __always_inline void __idtentry_exit(struct pt_regs *regs) · line 657
Quoted 10 lines
arch/x86/entry/common.c +10
-0+idtentry_enter (regs); + old_regs =set_irq_regs (regs); + +if (!irq_needs_irq_stack (regs)) { +instrumentation_begin (); +__xen_pv_evtchn_do_upcall (); +instrumentation_end (); + }else { +run_on_irqstack (__xen_pv_evtchn_do_upcall,NULL ); + }
Shouldn't this be:
instrumentation_begin();
if (!irq_needs_irq_stack(...))
__blah();
else
run_on_irqstack(__blah, NULL);instrumentation_end();
or even:
instrumentation_begin();run_on_irqstack_if_needed(__blah, NULL);instrumentation_end();
****** BUT *******
I think this is all arse-backwards. This is a giant mess designed to pretend we support preemption and to emulate normal preemption in a non-preemptible kernel. I propose one to two massive cleanups:
A: Just delete all of this code. Preemptible hypercalls on non-preempt kernels will still process interrupts but won't get preempted. If you want preemption, compile with preemption.
B: Turn this thing around. Specifically, in the one and only case we care about, we know pretty much exactly what context we got this entry
in: we're running in a schedulable context doing an explicitly preemptible hypercall, and we have RIP pointing at a SYSCALL instruction (presumably, but we shouldn't bet on it) in the hypercall page. Ideally we would change the Xen PV ABI so the hypercall would return something like EAGAIN instead of auto-restarting and we could ditch this mess entirely. But the ABI seems to be set in stone or at least in molasses, so how about just:
idt_entry(exit(regs));
if (inhcall && need_resched())
schedule();
Off the top of my head, I don't see any reason this wouldn't work, and it's a heck of a lot cleaner. Possibly it should really be:
if (inhcall) {
if (!WARN_ON(regs->ip not in hypercall page))
cond_resched(); }
Re: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 10 lines · Andy Lutomirski
Quoted 9 lines · Thomas Gleixner
arch/x86/entry/common.c +5
-0@@ -573,6 +578,16 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs)instrumentation_end ();return ; } + }else if (IS_ENABLED (CONFIG_XEN_PV )) { +if (preempt_hcall) { +/* See CONFIG_PREEMPTION above */ +instrumentation_begin (); +rcu_irq_exit_preempt ();
Quoted 8 lines
arch/x86/entry/common.c static __always_inline void __idtentry_exit(struct pt_regs *regs) · line 590
Quoted 5 lines
arch/x86/entry/common.c +5
-0+xen_maybe_preempt_hcall (); +trace_hardirqs_on (); +instrumentation_end (); +return ; + }Ewwwww! This shouldn't be taken as a NAK -- it's just an expression of disgust.
I'm really not proud of it, but that was the least horrible thing I could come up with.
Quoted 9 lines
Shouldn't this be:
instrumentation_begin();
if (!irq_needs_irq_stack(...))
__blah();
else
run_on_irqstack(__blah, NULL);instrumentation_end();
Quoted 5 lines
or even:
instrumentation_begin();run_on_irqstack_if_needed(__blah, NULL);instrumentation_end();
Yeah. In that case the instrumentation markers are not required as they will be inside the run....() function.
Quoted 9 lines
****** BUT *******
I think this is all arse-backwards. This is a giant mess designed to pretend we support preemption and to emulate normal preemption in a non-preemptible kernel. I propose one to two massive cleanups:
A: Just delete all of this code. Preemptible hypercalls on non-preempt kernels will still process interrupts but won't get preempted. If you want preemption, compile with preemption.
I'm happy to do so, but the XEN folks might have opinions on that :)
Quoted 10 lines
B: Turn this thing around. Specifically, in the one and only case we care about, we know pretty much exactly what context we got this entry
in: we're running in a schedulable context doing an explicitly preemptible hypercall, and we have RIP pointing at a SYSCALL instruction (presumably, but we shouldn't bet on it) in the hypercall page. Ideally we would change the Xen PV ABI so the hypercall would return something like EAGAIN instead of auto-restarting and we could ditch this mess entirely. But the ABI seems to be set in stone or at least in molasses, so how about just:
Quoted 3 lines
idt_entry(exit(regs));
if (inhcall &&need_resched())
schedule();
Which brings you into the situation that you call schedule() from the point where we just moved it out. If we would go there we'd need to ensure that RCU is watching as well. idtentry_exit() might have it turned off ....
That's why I did it this way to keep the code flow exactly the same for all these exit variants.
Thanks,
tglx
Andy Lutomirski
maintainerRe: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 64 lines · Thomas Gleixner
Quoted 10 lines · Andy Lutomirski
Quoted 9 lines · Thomas Gleixner
arch/x86/entry/common.c +5
-0@@ -573,6 +578,16 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs)instrumentation_end ();return ; } + }else if (IS_ENABLED (CONFIG_XEN_PV )) { +if (preempt_hcall) { +/* See CONFIG_PREEMPTION above */ +instrumentation_begin (); +rcu_irq_exit_preempt ();Quoted 8 lines
arch/x86/entry/common.c static __always_inline void __idtentry_exit(struct pt_regs *regs) · line 590
Quoted 5 lines
arch/x86/entry/common.c +5
-0+xen_maybe_preempt_hcall (); +trace_hardirqs_on (); +instrumentation_end (); +return ; + }Ewwwww! This shouldn't be taken as a NAK -- it's just an expression of disgust.
I'm really not proud of it, but that was the least horrible thing I could come up with.
Quoted 9 lines
Shouldn't this be:
instrumentation_begin();
if (!irq_needs_irq_stack(...))
__blah();
else
run_on_irqstack(__blah, NULL);instrumentation_end();Quoted 5 lines
or even:
instrumentation_begin();run_on_irqstack_if_needed(__blah, NULL);instrumentation_end();Yeah. In that case the instrumentation markers are not required as they will be inside the run....() function.
Quoted 9 lines
****** BUT *******
I think this is all arse-backwards. This is a giant mess designed to pretend we support preemption and to emulate normal preemption in a non-preemptible kernel. I propose one to two massive cleanups:
A: Just delete all of this code. Preemptible hypercalls on non-preempt kernels will still process interrupts but won't get preempted. If you want preemption, compile with preemption.
I'm happy to do so, but the XEN folks might have opinions on that :)
Quoted 10 lines
B: Turn this thing around. Specifically, in the one and only case we care about, we know pretty much exactly what context we got this entry
in: we're running in a schedulable context doing an explicitly preemptible hypercall, and we have RIP pointing at a SYSCALL instruction (presumably, but we shouldn't bet on it) in the hypercall page. Ideally we would change the Xen PV ABI so the hypercall would return something like EAGAIN instead of auto-restarting and we could ditch this mess entirely. But the ABI seems to be set in stone or at least in molasses, so how about just:
Quoted 8 lines
Quoted 3 lines
idt_entry(exit(regs));
if (inhcall &&need_resched())
schedule();Which brings you into the situation that you call
schedule()from the point where we just moved it out. If we would go there we'd need to ensure that RCU is watching as well.idtentry_exit()might have it turned off ....
I don't think this is possible. Once you untangle all the wrappers, the call sites are effectively:
__this_cpu_write(xen_in_preemptible_hcall, true); CALL_NOSPEC to the hypercall page __this_cpu_write(xen_in_preemptible_hcall, false);
I think IF=1 when this happens, but I won't swear to it. RCU had better be watching.
As I understand it, the one and only situation Xen wants to handle is that an interrupt gets delivered during the hypercall. The hypervisor is too clever for its own good and deals with this by rewinding RIP to the beginning of whatever instruction did the hypercall and delivers the interrupt, and we end up in this handler. So, if this happens, the idea is to not only handle the interrupt but to schedule if scheduling would be useful.
So I don't think we need all this RCU magic. This really ought to be able to be simplified to:
idtentry_exit();
if (appropriate condition)
schedule();
Obviously we don't want to schedule if this is a nested entry, but we should be able to rule that out by checking that regs->flags & X86_EFLAGS_IF and by handling the percpu variable a little more intelligently. So maybe the right approach is:
bool in_preemptible_hcall = __this_cpu_read(xen_in_preemptible_hcall); __this_cpu_write(xen_in_preemptible_hcall, false); idtentry_enter(...);
do the acutal work;
idtentry_exit(...);
if (in_preemptible_hcall) {
assert regs->flags & X86_EFLAGS_IF;
assert that RCU is watching;
assert that we're on the thread stack;
assert whatever else we feel like asserting;
if (need_resched())
schedule(); }
__this_cpu_write(xen_in_preemptible_hcall, in_preemptible_hcall);
And now we don't have a special idtentry_exit() case just for Xen, and all the mess is entirely contained in the Xen PV code. And we need to mark all the preemptible hypercalls noinstr. Does this seem reasonable?
That being said, right now, with or without your patch, I think we're toast if the preemptible hypercall code gets traced. So maybe the right thing is to just drop all the magic preemption stuff from your patch and let the Xen maintainers submit something new (maybe like what I suggest above) if they want magic preemption back.
Jürgen Groß
maintainerRe: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 49 lines · Andy Lutomirski
Quoted 48 lines · Thomas Gleixner
Quoted 10 lines · Andy Lutomirski
Quoted 9 lines · Thomas Gleixner
arch/x86/entry/common.c +5
-0@@ -573,6 +578,16 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs)instrumentation_end ();return ; } + }else if (IS_ENABLED (CONFIG_XEN_PV )) { +if (preempt_hcall) { +/* See CONFIG_PREEMPTION above */ +instrumentation_begin (); +rcu_irq_exit_preempt ();Quoted 8 lines
arch/x86/entry/common.c static __always_inline void __idtentry_exit(struct pt_regs *regs) · line 590
Quoted 5 lines
arch/x86/entry/common.c +5
-0+xen_maybe_preempt_hcall (); +trace_hardirqs_on (); +instrumentation_end (); +return ; + }Ewwwww! This shouldn't be taken as a NAK -- it's just an expression of disgust.
I'm really not proud of it, but that was the least horrible thing I could come up with.
Quoted 9 lines
Shouldn't this be:
instrumentation_begin();
if (!irq_needs_irq_stack(...))
__blah();
else
run_on_irqstack(__blah, NULL);instrumentation_end();Quoted 5 lines
or even:
instrumentation_begin();run_on_irqstack_if_needed(__blah, NULL);instrumentation_end();Yeah. In that case the instrumentation markers are not required as they will be inside the run....() function.
Quoted 6 lines
****** BUT *******
I think this is all arse-backwards. This is a giant mess designed to pretend we support preemption and to emulate normal preemption in a non-preemptible kernel. I propose one to two massive cleanups:
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
A: Just delete all of this code. Preemptible hypercalls on non-preempt kernels will still process interrupts but won't get preempted. If you want preemption, compile with preemption.
I'm happy to do so, but the XEN folks might have opinions on that :)
Indeed. :-)
Quoted 23 lines
Quoted 11 lines
Quoted 11 lines
B: Turn this thing around. Specifically, in the one and only case we care about, we know pretty much exactly what context we got this entry
in: we're running in a schedulable context doing an explicitly preemptible hypercall, and we have RIP pointing at a SYSCALL instruction (presumably, but we shouldn't bet on it) in the hypercall page. Ideally we would change the Xen PV ABI so the hypercall would return something like EAGAIN instead of auto-restarting and we could ditch this mess entirely. But the ABI seems to be set in stone or at least in molasses, so how about just:Quoted 8 lines
Quoted 3 lines
idt_entry(exit(regs));
if (inhcall &&need_resched())
schedule();Which brings you into the situation that you call
schedule()from the point where we just moved it out. If we would go there we'd need to ensure that RCU is watching as well.idtentry_exit()might have it turned off ....I don't think this is possible. Once you untangle all the wrappers, the call sites are effectively:
Quoted 6 lines
__this_cpu_write(xen_in_preemptible_hcall, true);CALL_NOSPECto the hypercall page__this_cpu_write(xen_in_preemptible_hcall, false);I think IF=1 when this happens, but I won't swear to it. RCU had better be watching.
Preemptible hypercalls are never done with interrupts off. To be more
precise: they are only ever done during ioctl() processing.
I can add an ASSERT() to xen_preemptible_hcall_begin() if you want.
Quoted 8 lines
As I understand it, the one and only situation Xen wants to handle is that an interrupt gets delivered during the hypercall. The hypervisor is too clever for its own good and deals with this by rewinding RIP to the beginning of whatever instruction did the hypercall and delivers the interrupt, and we end up in this handler. So, if this happens, the idea is to not only handle the interrupt but to schedule if scheduling would be useful.
Correct. More precise: the hypercalls in question can last very long (up to several seconds) and so they need to be interruptible. As said
before: the interface how this is done is horrible. :-(
Quoted 31 lines
So I don't think we need all this RCU magic. This really ought to be able to be simplified to:
idtentry_exit();if (appropriate condition)
schedule();Obviously we don't want to schedule if this is a nested entry, but we should be able to rule that out by checking that
regs->flags&X86_EFLAGS_IFand by handling the percpu variable a little more intelligently. So maybe the right approach is:bool
in_preemptible_hcall=__this_cpu_read(xen_in_preemptible_hcall);__this_cpu_write(xen_in_preemptible_hcall, false);idtentry_enter(...);do the acutal work;
idtentry_exit(...);if (in_preemptible_hcall) { assert regs->flags & X86_EFLAGS_IF; assert that RCU is watching; assert that we're on the thread stack; assert whatever else we feel like asserting; if (need_resched()) schedule(); }
Quoted 6 lines
__this_cpu_write(xen_in_preemptible_hcall, in_preemptible_hcall);And now we don't have a special
idtentry_exit()case just for Xen, and all the mess is entirely contained in the Xen PV code. And we need to mark all the preemptible hypercalls noinstr. Does this seem reasonable?
From my point of view this sounds fine.
Quoted 7 lines
That being said, right now, with or without your patch, I think we're toast if the preemptible hypercall code gets traced. So maybe the right thing is to just drop all the magic preemption stuff from your patch and let the Xen maintainers submit something new (maybe like what I suggest above) if they want magic preemption back.
I'd prefer to not break preemptible hypercall in between.
IMO the patch should be modified along your suggestion. I'd be happy to test it.
Juergen
Re: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 48 lines · Jürgen Groß
Quoted 26 lines · Andy Lutomirski
Quoted 12 lines · Thomas Gleixner
Quoted 10 lines · Andy Lutomirski
B: Turn this thing around. Specifically, in the one and only case we care about, we know pretty much exactly what context we got this entry
in: we're running in a schedulable context doing an explicitly preemptible hypercall, and we have RIP pointing at a SYSCALL instruction (presumably, but we shouldn't bet on it) in the hypercall page. Ideally we would change the Xen PV ABI so the hypercall would return something like EAGAIN instead of auto-restarting and we could ditch this mess entirely. But the ABI seems to be set in stone or at least in molasses, so how about just:Quoted 8 lines
Quoted 3 lines
idt_entry(exit(regs));
if (inhcall &&need_resched())
schedule();Which brings you into the situation that you call
schedule()from the point where we just moved it out. If we would go there we'd need to ensure that RCU is watching as well.idtentry_exit()might have it turned off ....I don't think this is possible. Once you untangle all the wrappers, the call sites are effectively:
Quoted 6 lines
__this_cpu_write(xen_in_preemptible_hcall, true);CALL_NOSPECto the hypercall page__this_cpu_write(xen_in_preemptible_hcall, false);I think IF=1 when this happens, but I won't swear to it. RCU had better be watching.
Preemptible hypercalls are never done with interrupts off. To be more
precise: they are only ever done duringioctl()processing.I can add an
ASSERT()toxen_preemptible_hcall_begin()if you want.Quoted 8 lines
As I understand it, the one and only situation Xen wants to handle is that an interrupt gets delivered during the hypercall. The hypervisor is too clever for its own good and deals with this by rewinding RIP to the beginning of whatever instruction did the hypercall and delivers the interrupt, and we end up in this handler. So, if this happens, the idea is to not only handle the interrupt but to schedule if scheduling would be useful.
Quoted 3 lines
Correct. More precise: the hypercalls in question can last very long (up to several seconds) and so they need to be interruptible. As said
before: the interface how this is done is horrible. :-(
Forget seconds. DOMCTL_domain_kill gets to ~14 minutes for a 2TB domain.
The reason for the existing logic is to be able voluntarily preempt.
It doesn't need to remain the way it is, but some adequate form of pre-emption does need to stay.
~Andrew
Re: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
In reply to Andy LutomirskiQuoted 24 lines · Andy Lutomirski
Quoted 4 lines · Thomas Gleixner
Which brings you into the situation that you call
schedule()from the point where we just moved it out. If we would go there we'd need to ensure that RCU is watching as well.idtentry_exit()might have it turned off ....I don't think this is possible. Once you untangle all the wrappers, the call sites are effectively:
__this_cpu_write(xen_in_preemptible_hcall, true);CALL_NOSPECto the hypercall page__this_cpu_write(xen_in_preemptible_hcall, false);I think IF=1 when this happens, but I won't swear to it. RCU had better be watching.
As I understand it, the one and only situation Xen wants to handle is that an interrupt gets delivered during the hypercall. The hypervisor is too clever for its own good and deals with this by rewinding RIP to the beginning of whatever instruction did the hypercall and delivers the interrupt, and we end up in this handler. So, if this happens, the idea is to not only handle the interrupt but to schedule if scheduling would be useful.
Quoted 7 lines
So I don't think we need all this RCU magic. This really ought to be able to be simplified to:
idtentry_exit();if (appropriate condition)
schedule();
This is exactly the kind of tinkering which causes all kinds of trouble.
idtentry_exit()
if (user_mode(regs)) {
prepare_exit_to_usermode(regs);
} else if (regs->flags & X86_EFLAGS_IF) {
/* Check kernel preemption, if enabled */
if (IS_ENABLED(CONFIG_PREEMPTION)) {
....
}
instrumentation_begin();
/* Tell the tracer that IRET will enable interrupts */
trace_hardirqs_on_prepare();
lockdep_hardirqs_on_prepare(CALLER_ADDR0);
instrumentation_end();
rcu_irq_exit();
lockdep_hardirqs_on(CALLER_ADDR0);
} else {
/* IRQ flags state is correct already. Just tell RCU */
rcu_irq_exit();
}So in case IF is set then this already told the tracer and lockdep that interrupts are enabled. And contrary to the ugly version this exit path does not use rcu_irq_exit_preempt() which is there to warn about crappy RCU state when trying to schedule.
So we went great length to sanitize _all_ of this and make it consistent just to say: screw it for that xen thingy.
The extra checks and extra warnings for scheduling come with the guarantee to bitrot when idtentry_exit() or any logic invoked from there is changed. It's going to look like this:
/*
* If the below causes problems due to inconsistent state
* or out of sync sanity checks, please complain to
* [email protected] directly.
*/
idtentry_exit();
if (user_mode(regs) || !(regs->flags & X86_FlAGS_IF))
return;
if (!__this_cpu_read(xen_in_preemptible_hcall))
return;
rcu_sanity_check_for_preemption();
if (need_resched()) {
instrumentation_begin();
xen_maybe_preempt_hcall();
trace_hardirqs_on();
instrumentation_end();
} Of course you need the extra rcu_sanity_check_for_preemption() function just for this muck.
That's a true win on all ends? I don't think so.
Thanks,
tglx
Andy Lutomirski
maintainerRe: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 91 lines · Thomas Gleixner
Quoted 24 lines · Andy Lutomirski
Quoted 4 lines · Thomas Gleixner
Which brings you into the situation that you call
schedule()from the point where we just moved it out. If we would go there we'd need to ensure that RCU is watching as well.idtentry_exit()might have it turned off ....I don't think this is possible. Once you untangle all the wrappers, the call sites are effectively:
__this_cpu_write(xen_in_preemptible_hcall, true);CALL_NOSPECto the hypercall page__this_cpu_write(xen_in_preemptible_hcall, false);I think IF=1 when this happens, but I won't swear to it. RCU had better be watching.
As I understand it, the one and only situation Xen wants to handle is that an interrupt gets delivered during the hypercall. The hypervisor is too clever for its own good and deals with this by rewinding RIP to the beginning of whatever instruction did the hypercall and delivers the interrupt, and we end up in this handler. So, if this happens, the idea is to not only handle the interrupt but to schedule if scheduling would be useful.
Quoted 7 lines
So I don't think we need all this RCU magic. This really ought to be able to be simplified to:
idtentry_exit();if (appropriate condition)
schedule();This is exactly the kind of tinkering which causes all kinds of trouble.
idtentry_exit()if (user_mode(regs)) { prepare_exit_to_usermode(regs); } else if (regs->flags & X86_EFLAGS_IF) { /* Check kernel preemption, if enabled */ if (IS_ENABLED(CONFIG_PREEMPTION)) { .... } instrumentation_begin(); /* Tell the tracer that IRET will enable interrupts */ trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); instrumentation_end(); rcu_irq_exit(); lockdep_hardirqs_on(CALLER_ADDR0); } else { /* IRQ flags state is correct already. Just tell RCU */ rcu_irq_exit(); }So in case IF is set then this already told the tracer and lockdep that interrupts are enabled. And contrary to the ugly version this exit path does not use
rcu_irq_exit_preempt()which is there to warn about crappy RCU state when trying to schedule.So we went great length to sanitize _all_ of this and make it consistent just to say: screw it for that xen thingy.
The extra checks and extra warnings for scheduling come with the guarantee to bitrot when
idtentry_exit()or any logic invoked from there is changed. It's going to look like this:/* * If the below causes problems due to inconsistent state * or out of sync sanity checks, please complain to * [email protected] directly. */ idtentry_exit(); if (user_mode(regs) || !(regs->flags & X86_FlAGS_IF)) return; if (!__this_cpu_read(xen_in_preemptible_hcall)) return; rcu_sanity_check_for_preemption(); if (need_resched()) { instrumentation_begin(); xen_maybe_preempt_hcall(); trace_hardirqs_on(); instrumentation_end(); }
Quoted 4 lines
Of course you need the extra
rcu_sanity_check_for_preemption()function just for this muck.That's a true win on all ends? I don't think so.
Hmm, fair enough. I guess the IRQ tracing messes a bunch of this logic up.
Let's keep your patch as is and consider cleanups later. One approach might be to make this work more like extable handling: instead of trying to schedule from inside the interrupt handler here, patch up RIP and perhaps some other registers and let the actual Xen code just do cond_resched(). IOW, try to make this work the way it always should have:
int ret;
do {
ret = issue_the_hypercall();
cond_resched();
} while (ret == EAGAIN);
Andy Lutomirski
maintainerRe: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 107 lines · Andy Lutomirski
Quoted 91 lines · Thomas Gleixner
Quoted 24 lines · Andy Lutomirski
Quoted 4 lines · Thomas Gleixner
Which brings you into the situation that you call
schedule()from the point where we just moved it out. If we would go there we'd need to ensure that RCU is watching as well.idtentry_exit()might have it turned off ....I don't think this is possible. Once you untangle all the wrappers, the call sites are effectively:
__this_cpu_write(xen_in_preemptible_hcall, true);CALL_NOSPECto the hypercall page__this_cpu_write(xen_in_preemptible_hcall, false);I think IF=1 when this happens, but I won't swear to it. RCU had better be watching.
As I understand it, the one and only situation Xen wants to handle is that an interrupt gets delivered during the hypercall. The hypervisor is too clever for its own good and deals with this by rewinding RIP to the beginning of whatever instruction did the hypercall and delivers the interrupt, and we end up in this handler. So, if this happens, the idea is to not only handle the interrupt but to schedule if scheduling would be useful.
Quoted 7 lines
So I don't think we need all this RCU magic. This really ought to be able to be simplified to:
idtentry_exit();if (appropriate condition)
schedule();This is exactly the kind of tinkering which causes all kinds of trouble.
idtentry_exit()if (user_mode(regs)) { prepare_exit_to_usermode(regs); } else if (regs->flags & X86_EFLAGS_IF) { /* Check kernel preemption, if enabled */ if (IS_ENABLED(CONFIG_PREEMPTION)) { .... } instrumentation_begin(); /* Tell the tracer that IRET will enable interrupts */ trace_hardirqs_on_prepare(); lockdep_hardirqs_on_prepare(CALLER_ADDR0); instrumentation_end(); rcu_irq_exit(); lockdep_hardirqs_on(CALLER_ADDR0); } else { /* IRQ flags state is correct already. Just tell RCU */ rcu_irq_exit(); }So in case IF is set then this already told the tracer and lockdep that interrupts are enabled. And contrary to the ugly version this exit path does not use
rcu_irq_exit_preempt()which is there to warn about crappy RCU state when trying to schedule.So we went great length to sanitize _all_ of this and make it consistent just to say: screw it for that xen thingy.
The extra checks and extra warnings for scheduling come with the guarantee to bitrot when
idtentry_exit()or any logic invoked from there is changed. It's going to look like this:/* * If the below causes problems due to inconsistent state * or out of sync sanity checks, please complain to * [email protected] directly. */ idtentry_exit(); if (user_mode(regs) || !(regs->flags & X86_FlAGS_IF)) return; if (!__this_cpu_read(xen_in_preemptible_hcall)) return; rcu_sanity_check_for_preemption(); if (need_resched()) { instrumentation_begin(); xen_maybe_preempt_hcall(); trace_hardirqs_on(); instrumentation_end(); }Quoted 4 lines
Of course you need the extra
rcu_sanity_check_for_preemption()function just for this muck.That's a true win on all ends? I don't think so.
Hmm, fair enough. I guess the IRQ tracing messes a bunch of this logic up.
Let's keep your patch as is and consider cleanups later. One approach might be to make this work more like extable handling: instead of trying to schedule from inside the interrupt handler here, patch up RIP and perhaps some other registers and let the actual Xen code just do
cond_resched(). IOW, try to make this work the way it always should have:
Quoted 5 lines
int ret;
do {
ret =issue_the_hypercall();
cond_resched();
} while (ret == EAGAIN);
Andrew Cooper pointed out that there is too much magic in Xen for this to work. So never mind.
Re: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 2 lines · Andy Lutomirski
Andrew Cooper pointed out that there is too much magic in Xen for this to work. So never mind.
:)
But you made me stare more at that stuff and I came up with a way simpler solution. See below.
Thanks,
tglx
8<--------------
arch/x86/entry/common.c | 75 ++++++++++++++++++++++++++++++++++++++-- arch/x86/entry/entry_32.S | 17 ++++----- arch/x86/entry/entry_64.S | 22 +++-------- arch/x86/include/asm/idtentry.h | 13 ++++++ arch/x86/xen/setup.c | 4 +- arch/x86/xen/smp_pv.c | 3 + arch/x86/xen/xen-asm_32.S | 12 +++--- arch/x86/xen/xen-asm_64.S | 2 - arch/x86/xen/xen-ops.h | 1 drivers/xen/Makefile | 2 - drivers/xen/preempt.c | 42 ---------------------- 11 files changed, 115 insertions(+), 78 deletions(-)
arch/x86/entry/common.c +72 -3
@@ -27,6 +27,9 @@#include <linux/syscalls.h> #include <linux/uaccess.h> +#include <xen/xen-ops.h> +#include <xen/events.h> +#include <asm/desc.h> #include <asm/traps.h> #include <asm/vdso.h>
@@ -35,6 +38,7 @@#include <asm/nospec-branch.h> #include <asm/io_bitmap.h> #include <asm/syscall.h> +#include <asm/irq_stack.h> #define CREATE_TRACE_POINTS #include <trace/events/syscalls.h>
@@ -539,7 +543,8 @@ void noinstr idtentry_enter(struct pt_re } } -static __always_inline void __idtentry_exit(struct pt_regs *regs) +static __always_inline void __idtentry_exit(struct pt_regs *regs, +bool may_sched) {lockdep_assert_irqs_disabled ();
@@ -548,7 +553,7 @@ static __always_inline void __idtentry_eprepare_exit_to_usermode (regs); }else if (regs->flags &X86_EFLAGS_IF ) {/* Check kernel preemption, if enabled */ -if (IS_ENABLED (CONFIG_PREEMPTION )) { +if (IS_ENABLED (CONFIG_PREEMPTION ) || may_resched) {/* * This needs to be done very carefully. * idtentry_enter() invoked rcu_irq_enter(). This
@@ -612,5 +617,69 @@ static __always_inline void __idtentry_e*/ void noinstr idtentry_exit(struct pt_regs *regs) { -__idtentry_exit (regs); +__idtentry_exit (regs, false); +} + +#ifdef CONFIG_XEN_PV + +#ifndef CONFIG_PREEMPTION +/* +* Some hypercalls issued by the toolstack can take many 10s of +* seconds. Allow tasks running hypercalls via the privcmd driver to +* be voluntarily preempted even if full kernel preemption is +* disabled. +* +* Such preemptible hypercalls are bracketed by +* xen_preemptible_hcall_begin() and xen_preemptible_hcall_end() +* calls. +*/ +DEFINE_PER_CPU (bool, xen_in_preemptible_hcall); +EXPORT_SYMBOL_GPL (xen_in_preemptible_hcall); + +/* +* In case of scheduling the flag must be cleared and restored after +* returning from schedule as the task might move to a different CPU. +*/ +static __always_inline bool get_and_clear_inhcall(void) +{ +boot inhcall =__this_cpu_read (xen_in_preemptible_hcall); + +__this_cpu_write (xen_in_preemptible_hcall, false); +} + +static __always_inline void restore_inhcall(bool inhcall) +{ +__this_cpu_write (xen_in_preemptible_hcall, inhcall); +} +#else +static __always_inline bool get_and_clear_inhcall(void) { return false; } +static __always_inline void restore_inhcall(bool inhcall) { } +#endif + +static void __xen_pv_evtchn_do_upcall (void ) +{ +irq_enter_rcu (); +inc_irq_stat (irq_hv_callback_count); + +xen_hvm_evtchn_do_upcall (); + +irq_exit_rcu (); +} + +__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +bool inhcall; + +idtentry_enter (regs); + old_regs =set_irq_regs (regs); + +run_on_irqstack (__xen_pv_evtchn_do_upcall,NULL , regs); + +set_irq_regs (old_regs); + + inhcall =get_and_clear_inhcall (); +__idtentry_exit (regs, inhcall); +restore_inhcall (inhcall); } +#endif /* CONFIG_XEN_PV */
arch/x86/entry/entry_32.S +9 -8
@@ -1298,7 +1298,10 @@ SYM_CODE_END(native_iret) #endif #ifdef CONFIG_XEN_PV -SYM_FUNC_START(xen_hypervisor_callback) +/* + * See comment in entry_64.S for further explanation + */ +SYM_FUNC_START(exc_xen_hypervisor_callback) /* * Check to see if we got the event in the critical * region in xen_iret_direct, after we've reenabled
@@ -1315,14 +1318,11 @@ SYM_FUNC_START(xen_hypervisor_callback) pushl $-1 /* orig_ax = -1 => not a system call */ SAVE_ALL ENCODE_FRAME_POINTER - TRACE_IRQS_OFF + mov %esp, %eax - call xen_evtchn_do_upcall -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp ret_from_intr -SYM_FUNC_END(xen_hypervisor_callback) + call xen_pv_evtchn_do_upcall + jmp handle_exception_return +SYM_FUNC_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
@@ -1464,6 +1464,7 @@ SYM_CODE_START_LOCAL_NOALIGN(handle_exce movl %esp, %eax # pt_regs pointer CALL_NOSPEC edi +handle_exception_return: #ifdef CONFIG_VM86 movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS movb PT_CS(%esp), %al
arch/x86/entry/entry_64.S +7 -15
@@ -1067,10 +1067,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1 -#ifdef CONFIG_XEN_PV -idtentry 512 /* dummy */ hypervisor_callback xen_do_hypervisor_callback has_error_code=0 -#endif - /* * Reload gs selector with exception handling * edi: new selector
@@ -1158,9 +1154,10 @@ SYM_FUNC_END(asm_call_on_stack) * So, on entry to the handler we detect whether we interrupted an * existing activation in its critical region -- if so, we pop the current * activation and restart the handler using the previous one. + * + * C calling convention: exc_xen_hypervisor_callback(struct *pt_regs) */ -/* do_hypervisor_callback(struct *pt_regs) */ -SYM_CODE_START_LOCAL(xen_do_hypervisor_callback) +SYM_CODE_START_LOCAL(exc_xen_hypervisor_callback) /* * Since we don't modify %rdi, evtchn_do_upall(struct *pt_regs) will
@@ -1170,15 +1167,10 @@ SYM_CODE_START_LOCAL(xen_do_hypervisor_c movq %rdi, %rsp /* we don't return, adjust the stack frame */ UNWIND_HINT_REGS - ENTER_IRQ_STACK old_rsp=%r10 - call xen_evtchn_do_upcall - LEAVE_IRQ_STACK - -#ifndef CONFIG_PREEMPTION - call xen_maybe_preempt_hcall -#endif - jmp error_exit -SYM_CODE_END(xen_do_hypervisor_callback) + call xen_pv_evtchn_do_upcall + + jmp error_return +SYM_CODE_END(exc_xen_hypervisor_callback) /* * Hypervisor uses this for application faults while it executes.
arch/x86/include/asm/idtentry.h +13 -0
@@ -332,6 +332,13 @@ static __always_inline void __##func(str* This avoids duplicate defines and ensures that everything is consistent. */ +/* +* Dummy trap number so the low level ASM macro vector number checks do not +* match which results in emitting plain IDTENTRY stubs without bells and +* whistels. +*/ +#define X86_TRAP_OTHER 0xFFFF +/* Simple exception entry points. No hardware error code */ DECLARE_IDTENTRY (X86_TRAP_DE , exc_divide_error);DECLARE_IDTENTRY (X86_TRAP_OF , exc_overflow);
@@ -371,4 +378,10 @@ DECLARE_IDTENTRY_XEN(X86_TRAP_DB, debug)/* #DF */ DECLARE_IDTENTRY_DF (X86_TRAP_DF , exc_double_fault); +#ifdef CONFIG_XEN_PV +DECLARE_IDTENTRY (X86_TRAP_OTHER , exc_xen_hypervisor_callback); +#endif + +#undef X86_TRAP_OTHER +#endif
arch/x86/xen/setup.c +3 -1
@@ -20,6 +20,7 @@#include <asm/setup.h> #include <asm/acpi.h> #include <asm/numa.h> +#include <asm/idtentry.h> #include <asm/xen/hypervisor.h> #include <asm/xen/hypercall.h>
@@ -993,7 +994,8 @@ static void __init xen_pvmmu_arch_setup(HYPERVISOR_vm_assist (VMASST_CMD_enable, VMASST_TYPE_pae_extended_cr3); -if (register_callback (CALLBACKTYPE_event, xen_hypervisor_callback) || +if (register_callback (CALLBACKTYPE_event, + xen_asm_exc_xen_hypervisor_callback) ||register_callback (CALLBACKTYPE_failsafe, xen_failsafe_callback))BUG ();
arch/x86/xen/smp_pv.c +2 -1
@@ -27,6 +27,7 @@#include <asm/paravirt.h> #include <asm/desc.h> #include <asm/pgtable.h> +#include <asm/idtentry.h> #include <asm/cpu.h> #include <xen/interface/xen.h>
@@ -347,7 +348,7 @@ cpu_initialize_context(unsigned int cpu, ctxt->gs_base_kernel =per_cpu_offset (cpu);#endif ctxt->event_callback_eip = - (unsigned long )xen_hypervisor_callback; + (unsigned long )xen_asm_exc_xen_hypervisor_callback; ctxt->failsafe_callback_eip = (unsigned long )xen_failsafe_callback;per_cpu (xen_cr3, cpu) =__pa (swapper_pg_dir);
arch/x86/xen/xen-asm_32.S +7 -5
@@ -93,7 +93,7 @@ SYM_CODE_START(xen_iret) /* * If there's something pending, mask events again so we can - * jump back into xen_hypervisor_callback. Otherwise do not + * jump back into exc_xen_hypervisor_callback. Otherwise do not * touch XEN_vcpu_info_mask. */ jne 1f
@@ -113,7 +113,7 @@ SYM_CODE_START(xen_iret) * Events are masked, so jumping out of the critical region is * OK. */ - je xen_hypervisor_callback + je asm_exc_xen_hypervisor_callback 1: iret xen_iret_end_crit:
@@ -127,7 +127,7 @@ SYM_CODE_END(xen_iret) .globl xen_iret_start_crit, xen_iret_end_crit /* - * This is called by xen_hypervisor_callback in entry_32.S when it sees + * This is called by exc_xen_hypervisor_callback in entry_32.S when it sees * that the EIP at the time of interrupt was between * xen_iret_start_crit and xen_iret_end_crit. *
@@ -144,7 +144,7 @@ SYM_CODE_END(xen_iret) * eflags } * cs } nested exception info * eip } - * return address : (into xen_hypervisor_callback) + * return address : (into asm_exc_xen_hypervisor_callback) * * In order to deliver the nested exception properly, we need to discard the * nested exception frame such that when we handle the exception, we do it
@@ -152,7 +152,8 @@ SYM_CODE_END(xen_iret) * * The only caveat is that if the outer eax hasn't been restored yet (i.e. * it's still on stack), we need to restore its value here. - */ +*/ +.pushsection .noinstr.text, "ax" SYM_CODE_START(xen_iret_crit_fixup) /* * Paranoia: Make sure we're really coming from kernel space.
@@ -181,3 +182,4 @@ SYM_CODE_START(xen_iret_crit_fixup) 2: ret SYM_CODE_END(xen_iret_crit_fixup) +.popsection
arch/x86/xen/xen-asm_64.S +1 -1
@@ -54,7 +54,7 @@ xen_pv_trap asm_exc_simd_coprocessor_err #ifdef CONFIG_IA32_EMULATION xen_pv_trap entry_INT80_compat #endif -xen_pv_trap hypervisor_callback +xen_pv_trap asm_exc_xen_hypervisor_callback __INIT SYM_CODE_START(xen_early_idt_handler_array)
arch/x86/xen/xen-ops.h +0 -1
@@ -8,7 +8,6 @@#include <xen/xen-ops.h> /* These are code, but not functions. Defined in entry.S */ -extern const char xen_hypervisor_callback[];extern const char xen_failsafe_callback[];void xen_sysenter_target (void );
drivers/xen/Makefile +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_HOTPLUG_CPU) += cpu_hotplug.o -obj-y += grant-table.o features.o balloon.o manage.o preempt.o time.o +obj-y += grant-table.o features.o balloon.o manage.o time.o obj-y += mem-reservation.o obj-y += events/ obj-y += xenbus/
drivers/xen/preempt.c deleted +0 -42
42 lines not on this page · raw message
Andy Lutomirski
maintainerRe: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 9 lines · Thomas Gleixner
Quoted 2 lines · Andy Lutomirski
Andrew Cooper pointed out that there is too much magic in Xen for this to work. So never mind.
:)
But you made me stare more at that stuff and I came up with a way simpler solution. See below.
I like it, but I bet it can be even simpler if you do the tickle_whatever_paulmck_call_it() change:
Quoted 5 lines
arch/x86/entry/common.c +5
-0+__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +bool inhcall; +
arch/x86/entry/common.c static __always_inline void __idtentry_e · line 683
Quoted 10 lines
arch/x86/entry/common.c +10
-0+idtentry_enter (regs); + old_regs =set_irq_regs (regs); + +run_on_irqstack (__xen_pv_evtchn_do_upcall,NULL , regs); + +set_irq_regs (old_regs); + + inhcall =get_and_clear_inhcall (); +__idtentry_exit (regs, inhcall); +restore_inhcall (inhcall);
How about:
inhcall = get_and_clear_inhcall();
if (inhcall) {
if (!WARN_ON_ONCE((regs->flags & X86_EFLAGS_IF) || preempt_count()) {
local_irq_enable();
cond_resched();
local_irq_disable();
}
}
restore_inhcall(inhcall);
idtentry_exit(regs);This could probably be tidied up by having a xen_maybe_preempt() that does the inhcall and resched mess.
The point is that, with the tickle_nohz_ stuff, there is nothing actually preventing IRQ handlers from sleeping as long as they aren't on the IRQ stack and as long as the interrupted context was safe to sleep in.
--Andy
Re: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 32 lines · Andy Lutomirski
Quoted 9 lines · Thomas Gleixner
Quoted 2 lines · Andy Lutomirski
Andrew Cooper pointed out that there is too much magic in Xen for this to work. So never mind.
:)
But you made me stare more at that stuff and I came up with a way simpler solution. See below.
I like it, but I bet it can be even simpler if you do the
tickle_whatever_paulmck_call_it()change:Quoted 5 lines
arch/x86/entry/common.c +5
-0+__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +bool inhcall; +arch/x86/entry/common.c static __always_inline void __idtentry_e · line 683
Quoted 10 lines
arch/x86/entry/common.c +10
-0+idtentry_enter (regs); + old_regs =set_irq_regs (regs); + +run_on_irqstack (__xen_pv_evtchn_do_upcall,NULL , regs); + +set_irq_regs (old_regs); + + inhcall =get_and_clear_inhcall (); +__idtentry_exit (regs, inhcall); +restore_inhcall (inhcall);How about:
Quoted 6 lines
inhcall = get_and_clear_inhcall();
if (inhcall) {
if (!WARN_ON_ONCE((regs->flags & X86_EFLAGS_IF) || preempt_count()) {
local_irq_enable();
cond_resched();
local_irq_disable();This really want's to use preempt_schedule_irq() as the above is racy vs. need_resched().
Quoted 5 lines
}
}
restore_inhcall(inhcall);
idtentry_exit(regs);Quoted 7 lines
This could probably be tidied up by having a
xen_maybe_preempt()that does the inhcall and resched mess.The point is that, with the
tickle_nohz_stuff, there is nothing actually preventing IRQ handlers from sleeping as long as they aren't on the IRQ stack and as long as the interrupted context was safe to sleep in.
You still lose the debug checks. I'm working on it ...
Thanks,
tglx
Boris Ostrovsky
reviewerRe: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
In reply to Thomas Gleixnerarch/x86/entry/common.c static __always_inline void __idtentry_e · line 677
Quoted 9 lines · Thomas Gleixner
arch/x86/entry/common.c +9
-0+__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +bool inhcall; + +idtentry_enter (regs); + old_regs =set_irq_regs (regs); + +run_on_irqstack (__xen_pv_evtchn_do_upcall,NULL , regs);
We need to handle nested case (i.e. !irq_needs_irq_stack(), like in your original version). Moving get_and_clear_inhcall() up should prevent scheduling when this happens.
-boris
Quoted 7 lines
arch/x86/entry/common.c +6
-0+ +set_irq_regs (old_regs); + + inhcall =get_and_clear_inhcall (); +__idtentry_exit (regs, inhcall); +restore_inhcall (inhcall); }
Re: [patch V6 10/37] x86/entry: Switch XEN/PV hypercall entry to IDTENTRY
Quoted 17 lines · Boris Ostrovsky
arch/x86/entry/common.c static __always_inline void __idtentry_e · line 677
Quoted 9 lines · Thomas Gleixner
arch/x86/entry/common.c +9
-0+__visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs) +{ +struct pt_regs *old_regs; +bool inhcall; + +idtentry_enter (regs); + old_regs =set_irq_regs (regs); + +run_on_irqstack (__xen_pv_evtchn_do_upcall,NULL , regs);We need to handle nested case (i.e. !irq_needs_irq_stack(), like in your original version). Moving
get_and_clear_inhcall()up should prevent scheduling when this happens.
I locally changed run_on_irqstack() to do the magic checks and select the right one.
Thanks,
tglx
PATCH v6 11/37 x86/entry/64: Simplify idtentry_body
In reply to Thomas GleixnerAll C functions which do not have an error code have been converted to the new IDTENTRY interface which does not expect an error code in the arguments. Spare the XORL.
arch/x86/entry/entry_64.S +0 -2
@@ -531,8 +531,6 @@ SYM_CODE_END(spurious_entries_start) .if \has_error_code == 1 movq ORIG_RAX(%rsp), %rsi /* get error code into 2nd argument*/ movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ - .else - xorl %esi, %esi /* Clear the error code */ .endif .if \vector == X86_TRAP_PF
Re: [patch V6 11/37] x86/entry/64: Simplify idtentry_body
Quoted 5 lines · Thomas Gleixner
All C functions which do not have an error code have been converted to the new IDTENTRY interface which does not expect an error code in the arguments. Spare the XORL.
PATCH v6 12/37 x86/entry: Provide idtentry_entry/exit_cond_rcu()
In reply to Thomas GleixnerThe pagefault handler cannot use the regular idtentry_enter() because that invokes rcu_irq_enter() if the pagefault was caused in the kernel. Not a problem per se, but kernel side page faults can schedule which is not possible without invoking rcu_irq_exit().
Adding rcu_irq_exit() and a matching rcu_irq_enter() into the actual pagefault handling code would be possible, but not pretty either.
Provide idtentry_entry/exit_cond_rcu() which calls rcu_irq_enter() only when RCU is not watching. The conditional RCU enabling is a correctness
issue: A kernel page fault which hits a RCU idle reason can neither schedule nor is it likely to survive. But avoiding RCU warnings or RCU side effects is at least increasing the chance for useful debug output.
The function is also useful for implementing lightweight reschedule IPI and KVM posted interrupt IPI entry handling later.
arch/x86/entry/common.c +125 -17
@@ -515,6 +515,36 @@ SYSCALL_DEFINE0(ni_syscall)return -ENOSYS ; } +static __always_inline bool __idtentry_enter(struct pt_regs *regs, +bool cond_rcu) +{ +if (user_mode (regs)) { +enter_from_user_mode (); + }else { +if (!cond_rcu || !__rcu_is_watching ()) { +/* +* If RCU is not watching then the same careful +* sequence vs. lockdep and tracing is required. +*/ +lockdep_hardirqs_off (CALLER_ADDR0 ); +rcu_irq_enter (); +instrumentation_begin (); +trace_hardirqs_off_prepare (); +instrumentation_end (); +return true; + }else { +/* +* If RCU is watching then the combo function +* can be used. +*/ +instrumentation_begin (); +trace_hardirqs_off (); +instrumentation_end (); + } + } +return false; +} +/** * idtentry_enter - Handle state tracking on idtentry * @regs: Pointer to pt_regs of interrupted context
@@ -532,19 +562,60 @@ SYSCALL_DEFINE0(ni_syscall)*/ void noinstr idtentry_enter(struct pt_regs *regs) { -if (user_mode (regs)) { -enter_from_user_mode (); - }else { -lockdep_hardirqs_off (CALLER_ADDR0 ); -rcu_irq_enter (); -instrumentation_begin (); -trace_hardirqs_off_prepare (); -instrumentation_end (); - } +__idtentry_enter (regs, false); +} + +/** +* idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional +* RCU handling +* @regs: Pointer to pt_regs of interrupted context +* +* Invokes: +* - lockdep irqflag state tracking as low level ASM entry disabled +* interrupts. +* +* - Context tracking if the exception hit user mode. +* +* - The hardirq tracer to keep the state consistent as low level ASM +* entry disabled interrupts. +* +* For kernel mode entries the conditional RCU handling is useful for two +* purposes +* +* 1) Pagefaults: Kernel code can fault and sleep, e.g. on exec. This code +* is not in an RCU idle section. If rcu_irq_enter() would be invoked +* then nothing would invoke rcu_irq_exit() before scheduling. +* +* If the kernel faults in a RCU idle section then all bets are off +* anyway but at least avoiding a subsequent issue vs. RCU is helpful for +* debugging. +* +* 2) Scheduler IPI: To avoid the overhead of a regular idtentry vs. RCU +* and irq_enter() the IPI can be made lightweight if the tracepoints +* are not enabled. While the IPI functionality itself does not require +* RCU (folding preempt count) it still calls out into instrumentable +* functions, e.g. ack_APIC_irq(). The scheduler IPI can hit RCU idle +* sections, so RCU needs to be adjusted. For the fast path case, e.g. +* KVM kicking a vCPU out of guest mode this can be avoided because the +* IPI is handled after KVM reestablished kernel context including RCU. +* +* For user mode entries enter_from_user_mode() must be invoked to +* establish the proper context for NOHZ_FULL. Otherwise scheduling on exit +* would not be possible. +* +* Returns: True if RCU has been adjusted on a kernel entry +* False otherwise +* +* The return value must be fed into the rcu_exit argument of +* idtentry_exit_cond_rcu(). +*/ +bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs) +{ +return __idtentry_enter (regs, true); } static __always_inline void __idtentry_exit(struct pt_regs *regs, -bool preempt_hcall) +bool preempt_hcall,bool rcu_exit) {lockdep_assert_irqs_disabled ();
@@ -570,7 +641,12 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs,if (IS_ENABLED (CONFIG_DEBUG_ENTRY ))WARN_ON_ONCE (!on_thread_stack ());instrumentation_begin (); -rcu_irq_exit_preempt (); +/* +* Conditional for idtentry_exit_cond_rcu(), +* unconditional for all other users. +*/ +if (rcu_exit) +rcu_irq_exit_preempt ();if (need_resched ())preempt_schedule_irq ();/* Covers both tracing and lockdep */
@@ -602,11 +678,22 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs,trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (CALLER_ADDR0 );instrumentation_end (); -rcu_irq_exit (); +/* +* Conditional for idtentry_exit_cond_rcu(), unconditional +* for all other users. +*/ +if (rcu_exit) +rcu_irq_exit ();lockdep_hardirqs_on (CALLER_ADDR0 ); }else { -/* IRQ flags state is correct already. Just tell RCU */ -rcu_irq_exit (); +/* +* IRQ flags state is correct already. Just tell RCU. +* +* Conditional for idtentry_exit_cond_rcu(), unconditional +* for all other users. +*/ +if (rcu_exit) +rcu_irq_exit (); } }
@@ -627,7 +714,28 @@ static __always_inline void __idtentry_exit(struct pt_regs *regs,*/ void noinstr idtentry_exit(struct pt_regs *regs) { -__idtentry_exit (regs, false); +__idtentry_exit (regs, false, true); +} + +/** +* idtentry_exit_cond_rcu - Handle return from exception with conditional RCU +* handling +* @regs: Pointer to pt_regs (exception entry regs) +* @rcu_exit: Invoke rcu_irq_exit() if true +* +* Depending on the return target (kernel/user) this runs the necessary +* preemption and work checks if possible and reguired and returns to +* the caller with interrupts disabled and no further work pending. +* +* This is the last action before returning to the low level ASM code which +* just needs to return to the appropriate context. +* +* Counterpart to idtentry_enter_cond_rcu(). The return value of the entry +* function must be fed into the @rcu_exit argument. +*/ +void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit) +{ +__idtentry_exit (regs, false, rcu_exit); }#ifdef CONFIG_XEN_PV
@@ -659,11 +767,11 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct pt_regs *regs)set_irq_regs (old_regs);if (IS_ENABLED (CONFIG_PREEMPTION )) { -__idtentry_exit (regs, false); +__idtentry_exit (regs, false, true); }else {bool inhcall =__this_cpu_read (xen_in_preemptible_hcall); -__idtentry_exit (regs, inhcall &&need_resched ()); +__idtentry_exit (regs, inhcall &&need_resched (), true); } }#endif /* CONFIG_XEN_PV */
arch/x86/include/asm/idtentry.h +3 -0
@@ -10,6 +10,9 @@void idtentry_enter (struct pt_regs *regs);void idtentry_exit (struct pt_regs *regs); +bool idtentry_enter_cond_rcu (struct pt_regs *regs); +void idtentry_exit_cond_rcu (struct pt_regs *regs,bool rcu_exit); +/** * DECLARE_IDTENTRY - Declare functions for simple IDT entry points * No error code pushed by hardware
Andy Lutomirski
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 10 lines · Thomas Gleixner
The pagefault handler cannot use the regular
idtentry_enter()because that invokesrcu_irq_enter()if the pagefault was caused in the kernel. Not a problem per se, but kernel side page faults can schedule which is not possible without invokingrcu_irq_exit().Adding
rcu_irq_exit()and a matchingrcu_irq_enter()into the actual pagefault handling code would be possible, but not pretty either.
Quoted 8 lines
Provide idtentry_entry/exit_cond_rcu() which calls
rcu_irq_enter()only when RCU is not watching. The conditional RCU enabling is a correctness
issue: A kernel page fault which hits a RCU idle reason can neither schedule nor is it likely to survive. But avoiding RCU warnings or RCU side effects is at least increasing the chance for useful debug output.The function is also useful for implementing lightweight reschedule IPI and KVM posted interrupt IPI entry handling later.
Why is this conditional? That is, couldn't we do this for all idtentry_enter() calls instead of just for page faults? Evil things like NMI shouldn't go through this path at all.
--Andy
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 15 lines · Andy Lutomirski
Quoted 14 lines · Thomas Gleixner
The pagefault handler cannot use the regular
idtentry_enter()because that invokesrcu_irq_enter()if the pagefault was caused in the kernel. Not a problem per se, but kernel side page faults can schedule which is not possible without invokingrcu_irq_exit().Adding
rcu_irq_exit()and a matchingrcu_irq_enter()into the actual pagefault handling code would be possible, but not pretty either.Provide idtentry_entry/exit_cond_rcu() which calls
rcu_irq_enter()only when RCU is not watching. The conditional RCU enabling is a correctness
issue: A kernel page fault which hits a RCU idle reason can neither schedule nor is it likely to survive. But avoiding RCU warnings or RCU side effects is at least increasing the chance for useful debug output.
Quoted 6 lines
Quoted 2 lines
The function is also useful for implementing lightweight reschedule IPI and KVM posted interrupt IPI entry handling later.
Why is this conditional? That is, couldn't we do this for all
idtentry_enter()calls instead of just for page faults? Evil things like NMI shouldn't go through this path at all.
I thought about that, but then ended up with the conclusion that RCU might be unhappy, but my conclusion might be fundamentally wrong.
Paul?
Thanks,
tglx
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 19 lines · Thomas Gleixner
Quoted 18 lines · Andy Lutomirski
Quoted 16 lines · Thomas Gleixner
The pagefault handler cannot use the regular
idtentry_enter()because that invokesrcu_irq_enter()if the pagefault was caused in the kernel. Not a problem per se, but kernel side page faults can schedule which is not possible without invokingrcu_irq_exit().Adding
rcu_irq_exit()and a matchingrcu_irq_enter()into the actual pagefault handling code would be possible, but not pretty either.Provide idtentry_entry/exit_cond_rcu() which calls
rcu_irq_enter()only when RCU is not watching. The conditional RCU enabling is a correctness
issue: A kernel page fault which hits a RCU idle reason can neither schedule nor is it likely to survive. But avoiding RCU warnings or RCU side effects is at least increasing the chance for useful debug output.The function is also useful for implementing lightweight reschedule IPI and KVM posted interrupt IPI entry handling later.
Quoted 6 lines
Quoted 3 lines
Why is this conditional? That is, couldn't we do this for all
idtentry_enter()calls instead of just for page faults? Evil things like NMI shouldn't go through this path at all.I thought about that, but then ended up with the conclusion that RCU might be unhappy, but my conclusion might be fundamentally wrong.
It's about this:
rcu_nmi_enter()
{
if (!rcu_is_watching()) {
make it watch;
} else if (!in_nmi()) {
do_magic_nohz_dyntick_muck();
}So if we do all irq/system vector entries conditional then the do_magic() gets never executed. After that I got lost...
Thanks,
tglx
Andy Lutomirski
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 38 lines · Thomas Gleixner
Quoted 19 lines · Thomas Gleixner
Quoted 18 lines · Andy Lutomirski
Quoted 16 lines · Thomas Gleixner
The pagefault handler cannot use the regular
idtentry_enter()because that invokesrcu_irq_enter()if the pagefault was caused in the kernel. Not a problem per se, but kernel side page faults can schedule which is not possible without invokingrcu_irq_exit().Adding
rcu_irq_exit()and a matchingrcu_irq_enter()into the actual pagefault handling code would be possible, but not pretty either.Provide idtentry_entry/exit_cond_rcu() which calls
rcu_irq_enter()only when RCU is not watching. The conditional RCU enabling is a correctness
issue: A kernel page fault which hits a RCU idle reason can neither schedule nor is it likely to survive. But avoiding RCU warnings or RCU side effects is at least increasing the chance for useful debug output.The function is also useful for implementing lightweight reschedule IPI and KVM posted interrupt IPI entry handling later.
Quoted 6 lines
Quoted 3 lines
Why is this conditional? That is, couldn't we do this for all
idtentry_enter()calls instead of just for page faults? Evil things like NMI shouldn't go through this path at all.I thought about that, but then ended up with the conclusion that RCU might be unhappy, but my conclusion might be fundamentally wrong.
It's about this:
rcu_nmi_enter(){ if (!rcu_is_watching()) { make it watch; } else if (!in_nmi()) { do_magic_nohz_dyntick_muck(); }
Quoted 2 lines
So if we do all irq/system vector entries conditional then the
do_magic()gets never executed. After that I got lost...
I'm also baffled by that magic, but I'm also not suggesting doing this to *all* entries -- just the not-super-magic ones that use idtentry_enter().
Paul, what is this code actually trying to do?
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 12 lines · Andy Lutomirski
Quoted 11 lines · Thomas Gleixner
Thomas Gleixner <[email protected]> writes:
It's about this:
rcu_nmi_enter(){ if (!rcu_is_watching()) { make it watch; } else if (!in_nmi()) { do_magic_nohz_dyntick_muck(); }
Quoted 8 lines
Quoted 2 lines
So if we do all irq/system vector entries conditional then the
do_magic()gets never executed. After that I got lost...I'm also baffled by that magic, but I'm also not suggesting doing this to *all* entries -- just the not-super-magic ones that use
idtentry_enter().Paul, what is this code actually trying to do?
Citing Paul from IRC:
"The way things are right now, you can leave out the rcu_irq_enter()
if this is not a nohz_full CPU.
Or if this is a nohz_full CPU, and the tick is already enabled, in that case you could also leave out the rcu_irq_enter().
Or even if this is a nohz_full CPU and it does not have the tick enabled, if it has been in the kernel less than a few tens of milliseconds, still OK to avoid invoking rcu_irq_enter()
But my guess is that it would be a lot simpler to just always call it.
Hope that helps.
Thanks,
tglx
Andy Lutomirski
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 31 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Quoted 11 lines · Thomas Gleixner
Thomas Gleixner <[email protected]> writes:
It's about this:
rcu_nmi_enter(){ if (!rcu_is_watching()) { make it watch; } else if (!in_nmi()) { do_magic_nohz_dyntick_muck(); }Quoted 8 lines
Quoted 2 lines
So if we do all irq/system vector entries conditional then the
do_magic()gets never executed. After that I got lost...I'm also baffled by that magic, but I'm also not suggesting doing this to *all* entries -- just the not-super-magic ones that use
idtentry_enter().Paul, what is this code actually trying to do?
Citing Paul from IRC:
"The way things are right now, you can leave out the
rcu_irq_enter()
if this is not anohz_fullCPU.Or if this is a
nohz_fullCPU, and the tick is already enabled, in that case you could also leave out thercu_irq_enter().
Quoted 8 lines
Or even if this is a
nohz_fullCPU and it does not have the tick enabled, if it has been in the kernel less than a few tens of milliseconds, still OK to avoid invokingrcu_irq_enter()But my guess is that it would be a lot simpler to just always call it.
Hope that helps.
Maybe?
Unless I've missed something, the effect here is that #PF hitting in an RCU-watching context will skip rcu_irq_enter(), whereas all IRQs (because you converted them) as well as other faults and traps will call rcu_irq_enter().
Once upon a time, we did this horrible thing where, on entry from user mode, we would turn on interrupts while still in CONTEXT_USER, which means we could get an IRQ in an extended quiescent state. This means that the IRQ code had to end the EQS so that IRQ handlers could use RCU. But I killed this a few years ago -- x86 Linux now has a rule that, if IF=1, we are *not* in an EQS with the sole exception of the idle code.
In my dream world, we would never ever get IRQs while in an EQS -- we would do MWAIT with IF=0 and we would exit the EQS before taking the interrupt. But I guess we still need to support HLT, which means we have this mess.
But I still think we can plausibly get rid of the conditional. If we get an IRQ or (egads!) a fault in idle context, we'll have !__rcu_is_watching(), but, AFAICT, we also have preemption off. So it should be okay to do rcu_irq_enter(). OTOH, if we get an IRQ or a fault anywhere else, then we either have a severe bug in the RCU code itself and the RCU code faulted (in which case we get what we deserve) or RCU is watching and all is well. This means that the rule will be that, if preemption is on, it's fine to schedule inside an idtentry_begin()/idtentry_end() pair.
The remaining bit is just the urgent thing, and I don't understand what's going on. Paul, could we split out the urgent logic all by itself so that the IRQ handlers could do rcu_poke_urgent()? Or am I entirely misunderstanding its purpose?
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 56 lines · Andy Lutomirski
Quoted 31 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Quoted 11 lines · Thomas Gleixner
Thomas Gleixner <[email protected]> writes:
It's about this:
rcu_nmi_enter(){ if (!rcu_is_watching()) { make it watch; } else if (!in_nmi()) { do_magic_nohz_dyntick_muck(); }Quoted 8 lines
Quoted 2 lines
So if we do all irq/system vector entries conditional then the
do_magic()gets never executed. After that I got lost...I'm also baffled by that magic, but I'm also not suggesting doing this to *all* entries -- just the not-super-magic ones that use
idtentry_enter().Paul, what is this code actually trying to do?
Citing Paul from IRC:
"The way things are right now, you can leave out the
rcu_irq_enter()
if this is not anohz_fullCPU.Or if this is a
nohz_fullCPU, and the tick is already enabled, in that case you could also leave out thercu_irq_enter().Quoted 8 lines
Or even if this is a
nohz_fullCPU and it does not have the tick enabled, if it has been in the kernel less than a few tens of milliseconds, still OK to avoid invokingrcu_irq_enter()But my guess is that it would be a lot simpler to just always call it.
Hope that helps.
Maybe?
Unless I've missed something, the effect here is that #PF hitting in an RCU-watching context will skip
rcu_irq_enter(), whereas all IRQs (because you converted them) as well as other faults and traps will callrcu_irq_enter().Once upon a time, we did this horrible thing where, on entry from user mode, we would turn on interrupts while still in
CONTEXT_USER, which means we could get an IRQ in an extended quiescent state. This means that the IRQ code had to end the EQS so that IRQ handlers could use RCU. But I killed this a few years ago -- x86 Linux now has a rule that, if IF=1, we are *not* in an EQS with the sole exception of the idle code.
Quoted 6 lines
In my dream world, we would never ever get IRQs while in an EQS -- we would do MWAIT with IF=0 and we would exit the EQS before taking the interrupt. But I guess we still need to support HLT, which means we have this mess.
But I still think we can plausibly get rid of the conditional.
You mean the conditional in rcu_nmi_enter()? In a NO_HZ_FULL=n system, this becomes:
if (!rcu_is_watching()) {
make it watch;
} else if (!in_nmi()) {
instrumentation_begin();
if (tick_nohz_full_cpu(rdp->cpu) && ... {
do stuff
}
instrumentation_end();
}But tick_nohz_full_cpu() is compile-time-known false, so as long as the compiler can ditch the instrumentation_begin() and instrumentation_end(), the entire "else if" clause evaporates.
Quoted 3 lines
If we
get an IRQ or (egads!) a fault in idle context, we'll have !__rcu_is_watching(), but, AFAICT, we also have preemption off.
Or we could be early in the kernel-entry code or late in the kernel-exit code, but as far as I know, preemption is disabled on those code paths. As are interrupts, right? And interrupts are disabled on the portions of the CPU-hotplug code where RCU is not watching, if I recall correctly.
I am guessing that interrupts from userspace are not at issue here, but completeness and all that.
Quoted 7 lines
So it
should be okay to dorcu_irq_enter(). OTOH, if we get an IRQ or a fault anywhere else, then we either have a severe bug in the RCU code itself and the RCU code faulted (in which case we get what we deserve) or RCU is watching and all is well. This means that the rule will be that, if preemption is on, it's fine to schedule inside an idtentry_begin()/idtentry_end() pair.
On this, I must defer to you guys.
Quoted 4 lines
The remaining bit is just the urgent thing, and I don't understand what's going on. Paul, could we split out the urgent logic all by itself so that the IRQ handlers could do
rcu_poke_urgent()? Or am I entirely misunderstanding its purpose?
A nohz_full CPU does not enable the scheduling-clock interrupt upon entry to the kernel. Normally, this is fine because that CPU will very quickly exit back to nohz_full userspace execution, so that RCU will see the quiescent state, either by sampling it directly or by deducing the CPU's passage through that quiescent state by comparing with state that was captured earlier. The grace-period kthread notices the lack of a quiescent state and will eventually set ->rcu_urgent_qs to trigger this code.
But if the nohz_full CPU stays in the kernel for an extended time, perhaps due to OOM handling or due to processing of some huge I/O that hits in-memory buffers/cache, then RCU needs some way of detecting quiescent states on that CPU. This requires the scheduling-clock interrupt to be alive and well.
Are there other ways to get this done? But of course! RCU could for example use smp_call_function_single() or use workqueues to force execution onto that CPU and enable the tick that way. This gets a little involved in order to avoid deadlock, but if the added check in rcu_nmi_enter() is causing trouble, something can be arranged. Though that something would cause more latency excursions than does the current code.
Or did you have something else in mind?
Thanx, Paul
Andy Lutomirski
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 63 lines · Paul E. McKenney
Quoted 61 lines · Andy Lutomirski
Quoted 31 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Quoted 11 lines · Thomas Gleixner
Thomas Gleixner <[email protected]> writes:
It's about this:
rcu_nmi_enter(){ if (!rcu_is_watching()) { make it watch; } else if (!in_nmi()) { do_magic_nohz_dyntick_muck(); }Quoted 8 lines
Quoted 2 lines
So if we do all irq/system vector entries conditional then the
do_magic()gets never executed. After that I got lost...I'm also baffled by that magic, but I'm also not suggesting doing this to *all* entries -- just the not-super-magic ones that use
idtentry_enter().Paul, what is this code actually trying to do?
Citing Paul from IRC:
"The way things are right now, you can leave out the
rcu_irq_enter()
if this is not anohz_fullCPU.Or if this is a
nohz_fullCPU, and the tick is already enabled, in that case you could also leave out thercu_irq_enter().Quoted 8 lines
Or even if this is a
nohz_fullCPU and it does not have the tick enabled, if it has been in the kernel less than a few tens of milliseconds, still OK to avoid invokingrcu_irq_enter()But my guess is that it would be a lot simpler to just always call it.
Hope that helps.
Maybe?
Unless I've missed something, the effect here is that #PF hitting in an RCU-watching context will skip
rcu_irq_enter(), whereas all IRQs (because you converted them) as well as other faults and traps will callrcu_irq_enter().Once upon a time, we did this horrible thing where, on entry from user mode, we would turn on interrupts while still in
CONTEXT_USER, which means we could get an IRQ in an extended quiescent state. This means that the IRQ code had to end the EQS so that IRQ handlers could use RCU. But I killed this a few years ago -- x86 Linux now has a rule that, if IF=1, we are *not* in an EQS with the sole exception of the idle code.In my dream world, we would never ever get IRQs while in an EQS -- we would do MWAIT with IF=0 and we would exit the EQS before taking the interrupt. But I guess we still need to support HLT, which means we have this mess.
Quoted 4 lines
Quoted 1 line
But I still think we can plausibly get rid of the conditional.
You mean the conditional in
rcu_nmi_enter()? In a NO_HZ_FULL=n system, this becomes:
So, I meant the conditional in tglx's patch that makes page faults special.
Quoted 9 lines
Quoted 4 lines
If we
get an IRQ or (egads!) a fault in idle context, we'll have !__rcu_is_watching(), but, AFAICT, we also have preemption off.Or we could be early in the kernel-entry code or late in the kernel-exit code, but as far as I know, preemption is disabled on those code paths. As are interrupts, right? And interrupts are disabled on the portions of the CPU-hotplug code where RCU is not watching, if I recall correctly.
Interrupts are off in the parts of the entry/exit that RCU considers to be user mode. We can get various faults, although these should be either NMI-like or events that genuinely or effectively happened in user mode.
Quoted 24 lines
A
nohz_fullCPU does not enable the scheduling-clock interrupt upon entry to the kernel. Normally, this is fine because that CPU will very quickly exit back tonohz_fulluserspace execution, so that RCU will see the quiescent state, either by sampling it directly or by deducing the CPU's passage through that quiescent state by comparing with state that was captured earlier. The grace-period kthread notices the lack of a quiescent state and will eventually set ->rcu_urgent_qs to trigger this code.But if the
nohz_fullCPU stays in the kernel for an extended time, perhaps due to OOM handling or due to processing of some huge I/O that hits in-memory buffers/cache, then RCU needs some way of detecting quiescent states on that CPU. This requires the scheduling-clock interrupt to be alive and well.Are there other ways to get this done? But of course! RCU could for example use
smp_call_function_single()or use workqueues to force execution onto that CPU and enable the tick that way. This gets a little involved in order to avoid deadlock, but if the added check inrcu_nmi_enter()is causing trouble, something can be arranged. Though that something would cause more latency excursions than does the current code.
Quoted 1 line
Or did you have something else in mind?
I'm trying to understand when we actually need to call the function. Is it just the scheduling interrupt that's supposed to call rcu_irq_enter()? But the scheduling interrupt is off, so I'm confused.
Andy Lutomirski
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 5 lines · Andy Lutomirski
Quoted 3 lines · Paul E. McKenney
Quoted 1 line · Andy Lutomirski
On Tue, May 19, 2020 at 2:20 PM Thomas Gleixner <[email protected]> wrote:
First, the patch as you submitted it is Acked-by: Andy Lutomirski <[email protected]>. I think there are cleanups that should happen, but I think the patch is correct.
About cleanups, concretely: I think that everything that calls __idtenter_entry() is called in one of a small number of relatively sane states:
1. User mode. This is easy.
2. Kernel, RCU is watching, everything is sane. We don't actually need to do any RCU entry/exit pairs -- we should be okay with just a hypothetical RCU tickle (and IRQ tracing, etc). This variant can sleep after the entry part finishes if regs->flags & IF and no one turned off preemption.
3. Kernel, RCU is not watching, system was idle. This can only be an actual interrupt.
So maybe the code can change to:
if (user_mode(regs)) {
enter_from_user_mode();
} else {
if (!__rcu_is_watching()) {
/*
* If RCU is not watching then the same careful
* sequence vs. lockdep and tracing is required.
*
* This only happens for IRQs that hit the idle loop, and
* even that only happens if we aren't using the sane
* MWAIT-while-IF=0 mode.
*/
lockdep_hardirqs_off(CALLER_ADDR0);
rcu_irq_enter();
instrumentation_begin();
trace_hardirqs_off_prepare();
instrumentation_end();
return true;
} else {
/*
* If RCU is watching then the combo function
* can be used.
*/
instrumentation_begin();
trace_hardirqs_off();
rcu_tickle();
instrumentation_end();
}
}
return false;This is exactly what you have except that the cond_rcu part is gone and I added rcu_tickle().
Paul, the major change here is that if an IRQ hits normal kernel code (i.e. code where RCU is watching and we're not in an EQS), the IRQ won't call rcu_irq_enter() and rcu_irq_exit(). Instead it will call rcu_tickle() on entry and nothing on exit. Does that cover all the bases?
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 59 lines · Andy Lutomirski
Quoted 5 lines · Andy Lutomirski
Quoted 3 lines · Paul E. McKenney
Quoted 1 line · Andy Lutomirski
On Tue, May 19, 2020 at 2:20 PM Thomas Gleixner <[email protected]> wrote:
First, the patch as you submitted it is Acked-by: Andy Lutomirski <[email protected]>. I think there are cleanups that should happen, but I think the patch is correct.
About cleanups, concretely: I think that everything that calls
__idtenter_entry()is called in one of a small number of relatively sane states:1. User mode. This is easy.
2. Kernel, RCU is watching, everything is sane. We don't actually need to do any RCU entry/exit pairs -- we should be okay with just a hypothetical RCU tickle (and IRQ tracing, etc). This variant can sleep after the entry part finishes if
regs->flags& IF and no one turned off preemption.3. Kernel, RCU is not watching, system was idle. This can only be an actual interrupt.
So maybe the code can change to:
if (user_mode(regs)) { enter_from_user_mode(); } else { if (!__rcu_is_watching()) { /* * If RCU is not watching then the same careful * sequence vs. lockdep and tracing is required. * * This only happens for IRQs that hit the idle loop, and * even that only happens if we aren't using the sane * MWAIT-while-IF=0 mode. */ lockdep_hardirqs_off(CALLER_ADDR0); rcu_irq_enter(); instrumentation_begin(); trace_hardirqs_off_prepare(); instrumentation_end(); return true; } else { /* * If RCU is watching then the combo function * can be used. */ instrumentation_begin(); trace_hardirqs_off(); rcu_tickle(); instrumentation_end(); } } return false;
Quoted 8 lines
This is exactly what you have except that the
cond_rcupart is gone and I addedrcu_tickle().Paul, the major change here is that if an IRQ hits normal kernel code (i.e. code where RCU is watching and we're not in an EQS), the IRQ won't call
rcu_irq_enter()andrcu_irq_exit(). Instead it will callrcu_tickle()on entry and nothing on exit. Does that cover all the bases?
From an RCU viewpoint, yes, give or take my concerns about someone putting rcu_tickle() on entry and rcu_irq_exit() on exit. Perhaps I can bring some lockdep trickery to bear.
But I must defer to Thomas and Peter on the non-RCU/non-NO_HZ_FULL portions of this.
Thanx, Paul
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 10 lines · Paul E. McKenney
Quoted 5 lines · Andy Lutomirski
Paul, the major change here is that if an IRQ hits normal kernel code (i.e. code where RCU is watching and we're not in an EQS), the IRQ won't call
rcu_irq_enter()andrcu_irq_exit(). Instead it will callrcu_tickle()on entry and nothing on exit. Does that cover all the bases?From an RCU viewpoint, yes, give or take my concerns about someone putting
rcu_tickle()on entry andrcu_irq_exit()on exit. Perhaps I can bring some lockdep trickery to bear.
An surplus rcu_irq_exit() should already trigger alarms today.
Quoted 2 lines
But I must defer to Thomas and Peter on the non-RCU/non-NO_HZ_FULL portions of this.
I don't see a problem. Let me write that into actual testable patch form.
Thanks,
tglx
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Paul E. McKenney
Quoted 5 lines · Andy Lutomirski
Paul, the major change here is that if an IRQ hits normal kernel code (i.e. code where RCU is watching and we're not in an EQS), the IRQ won't call
rcu_irq_enter()andrcu_irq_exit(). Instead it will callrcu_tickle()on entry and nothing on exit. Does that cover all the bases?
Quoted 5 lines
Quoted 3 lines
From an RCU viewpoint, yes, give or take my concerns about someone putting
rcu_tickle()on entry andrcu_irq_exit()on exit. Perhaps I can bring some lockdep trickery to bear.An surplus
rcu_irq_exit()should already trigger alarms today.
Fair point!
Quoted 5 lines
Quoted 2 lines
But I must defer to Thomas and Peter on the non-RCU/non-NO_HZ_FULL portions of this.
I don't see a problem. Let me write that into actual testable patch form.
Here is the RCU part, with my current best guess for the commit log.
Please note that this is on top of my -rcu stack, so some adjustment will likely be needed to pull it underneath Joel's series that removes the special-purpose bits at the bottom of the ->dynticks counter.
But a starting point, anyway.
Thanx, Paul
------------------------------------------------------------------------
commit ca05838a9a1809fafee63f488a7be8b30e1c2a6a
Author: Paul E. McKenney <[email protected]> Date: Wed May 20 15:03:07 2020 -0700
rcu: Abstract out tickle_nohz_for_rcu() from rcu_nmi_enter()
This commit splits out the nohz_full scheduler-tick enabling from the rest of the rcu_nmi_enter() logic. This allows short exception handlers that interrupt kernel code regions that RCU is already watching to just invoke tickle_nohz_for_rcu() at exception entry instead of having to invoke rcu_nmi_enter() on entry and also rcu_nmi_exit() on all exit paths.
include/linux/hardirq.h +5 -0
@@ -14,6 +14,10 @@ extern bool synchronize_hardirq(unsigned int irq);#if defined(CONFIG_TINY_RCU ) +static inline void tickle_nohz_for_rcu (void ) +{ +} +static inline void rcu_nmi_enter (void ) { }
@@ -23,6 +27,7 @@ static inline void rcu_nmi_exit(void) }#else +extern void tickle_nohz_for_rcu (void );extern void rcu_nmi_enter (void );extern void rcu_nmi_exit (void );#endif
kernel/rcu/tree.c +64 -23
105 lines not on this page · raw message
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 23 lines · Paul E. McKenney
Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Paul E. McKenney
Quoted 5 lines · Andy Lutomirski
Paul, the major change here is that if an IRQ hits normal kernel code (i.e. code where RCU is watching and we're not in an EQS), the IRQ won't call
rcu_irq_enter()andrcu_irq_exit(). Instead it will callrcu_tickle()on entry and nothing on exit. Does that cover all the bases?Quoted 5 lines
Quoted 3 lines
From an RCU viewpoint, yes, give or take my concerns about someone putting
rcu_tickle()on entry andrcu_irq_exit()on exit. Perhaps I can bring some lockdep trickery to bear.An surplus
rcu_irq_exit()should already trigger alarms today.Fair point!
Quoted 5 lines
Quoted 2 lines
But I must defer to Thomas and Peter on the non-RCU/non-NO_HZ_FULL portions of this.
I don't see a problem. Let me write that into actual testable patch form.
Quoted 7 lines
Here is the RCU part, with my current best guess for the commit log.
Please note that this is on top of my -rcu stack, so some adjustment will likely be needed to pull it underneath Joel's series that removes the special-purpose bits at the bottom of the ->dynticks counter.
But a starting point, anyway.
Same patch, but with updated commit log based on IRC discussion with Andy.
Thanx, Paul
------------------------------------------------------------------------
commit 1771ea9fac5748d1424d9214c51b2f79cc1176b6
Author: Paul E. McKenney <[email protected]> Date: Wed May 20 15:03:07 2020 -0700
rcu: Abstract out tickle_nohz_for_rcu() from rcu_nmi_enter()
There will likely be exception handlers that can sleep, which rules out the usual approach of invoking rcu_nmi_enter() on entry and also rcu_nmi_exit() on all exit paths. However, the alternative approach of just not calling anything can prevent RCU from coaxing quiescent states from nohz_full CPUs that are looping in the kernel: RCU must instead IPI them explicitly. It would be better to enable the scheduler tick on such CPUs to interact with RCU in a lighter-weight manner, and this enabling is one of the things that rcu_nmi_enter() currently does.
What is needed is something that helps RCU coax quiescent states while not preventing subsequent sleeps. This commit therefore splits out the nohz_full scheduler-tick enabling from the rest of the rcu_nmi_enter() logic into a new function named tickle_nohz_for_rcu().
include/linux/hardirq.h +5 -0
@@ -14,6 +14,10 @@ extern bool synchronize_hardirq(unsigned int irq);#if defined(CONFIG_TINY_RCU ) +static inline void tickle_nohz_for_rcu (void ) +{ +} +static inline void rcu_nmi_enter (void ) { }
@@ -23,6 +27,7 @@ static inline void rcu_nmi_exit(void) }#else +extern void tickle_nohz_for_rcu (void );extern void rcu_nmi_enter (void );extern void rcu_nmi_exit (void );#endif
kernel/rcu/tree.c +64 -23
105 lines not on this page · raw message
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 3 lines · Paul E. McKenney
On Wed, May 20, 2020 at 03:15:31PM -0700, Paul E. McKenney wrote:
Same patch, but with updated commit log based on IRC discussion with Andy.
Fun. I came up with the same thing before going to bed. Just that I named the function differently: rcu_irq_enter_check_tick()
Quoted 10 lines
#if
defined(CONFIG_TINY_RCU)include/linux/hardirq.h +4
-0+static inline void tickle_nohz_for_rcu (void ) +{ +} +static inline void rcu_nmi_enter (void ) { }
include/linux/hardirq.h static inline void rcu_nmi_exit(void) · line 30
Quoted 4 lines
include/linux/hardirq.h +1
-0@@ -23,6 +27,7 @@ static inline void rcu_nmi_exit(void) }#else +extern void tickle_nohz_for_rcu (void );
And I made this a NOP for for !NOHZ_FULL systems and avoided the call if context tracking is not enabled at boot.
void __rcu_irq_enter_check_tick(void);
static inline void rcu_irq_enter_check_tick(void)
{
if (context_tracking_enabled())
__rcu_irq_enter_check_tick();
}Thanks,
tglx
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 7 lines · Thomas Gleixner
Quoted 3 lines · Paul E. McKenney
On Wed, May 20, 2020 at 03:15:31PM -0700, Paul E. McKenney wrote:
Same patch, but with updated commit log based on IRC discussion with Andy.Fun. I came up with the same thing before going to bed. Just that I named the function differently:
rcu_irq_enter_check_tick()
I am good with that name.
Quoted 18 lines
Quoted 10 lines
#if
defined(CONFIG_TINY_RCU)include/linux/hardirq.h +4
-0+static inline void tickle_nohz_for_rcu (void ) +{ +} +static inline void rcu_nmi_enter (void ) { }include/linux/hardirq.h static inline void rcu_nmi_exit(void) · line 30
Quoted 4 lines
include/linux/hardirq.h +1
-0@@ -23,6 +27,7 @@ static inline void rcu_nmi_exit(void) }#else +extern void tickle_nohz_for_rcu (void );And I made this a NOP for for !NOHZ_FULL systems and avoided the call if context tracking is not enabled at boot.
Quoted 7 lines
void
__rcu_irq_enter_check_tick(void);static inline void
rcu_irq_enter_check_tick(void){ if (context_tracking_enabled()) __rcu_irq_enter_check_tick(); }
That certainly is a better approach!
So let's go with your version. But could you please post it, just in case reviewing an alternative version causes me to spot something?
Thanx, Paul
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 12 lines · Paul E. McKenney
Quoted 10 lines · Thomas Gleixner
And I made this a NOP for for !NOHZ_FULL systems and avoided the call if context tracking is not enabled at boot.
void
__rcu_irq_enter_check_tick(void);static inline void
rcu_irq_enter_check_tick(void){ if (context_tracking_enabled()) __rcu_irq_enter_check_tick(); }
Quoted 4 lines
That certainly is a better approach!
So let's go with your version. But could you please post it, just in case reviewing an alternative version causes me to spot something?
I'm just testing the complete rework of this series and will post it if it survives smoke testing.
Thanks,
tglx
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 13 lines · Thomas Gleixner
Quoted 12 lines · Paul E. McKenney
Quoted 10 lines · Thomas Gleixner
And I made this a NOP for for !NOHZ_FULL systems and avoided the call if context tracking is not enabled at boot.
void
__rcu_irq_enter_check_tick(void);static inline void
rcu_irq_enter_check_tick(void){ if (context_tracking_enabled()) __rcu_irq_enter_check_tick(); }
Quoted 7 lines
Quoted 4 lines
That certainly is a better approach!
So let's go with your version. But could you please post it, just in case reviewing an alternative version causes me to spot something?
I'm just testing the complete rework of this series and will post it if it survives smoke testing.
Fair enough!
Thanx, Paul
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
In reply to Andy LutomirskiQuoted 33 lines · Andy Lutomirski
So maybe the code can change to:
if (user_mode(regs)) { enter_from_user_mode(); } else { if (!__rcu_is_watching()) { /* * If RCU is not watching then the same careful * sequence vs. lockdep and tracing is required. * * This only happens for IRQs that hit the idle loop, and * even that only happens if we aren't using the sane * MWAIT-while-IF=0 mode. */ lockdep_hardirqs_off(CALLER_ADDR0); rcu_irq_enter(); instrumentation_begin(); trace_hardirqs_off_prepare(); instrumentation_end(); return true; } else { /* * If RCU is watching then the combo function * can be used. */ instrumentation_begin(); trace_hardirqs_off(); rcu_tickle(); instrumentation_end(); } } return false;
Quoted 8 lines
This is exactly what you have except that the
cond_rcupart is gone and I addedrcu_tickle().Paul, the major change here is that if an IRQ hits normal kernel code (i.e. code where RCU is watching and we're not in an EQS), the IRQ won't call
rcu_irq_enter()andrcu_irq_exit(). Instead it will callrcu_tickle()on entry and nothing on exit. Does that cover all the bases?
Fine with me, but the final vote needs to come from Paul and Joel.
Thanks,
tglx
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
In reply to Andy LutomirskiQuoted 32 lines · Andy Lutomirski
On Wed, May 20, 2020 at 8:36 AM Andy Lutomirski <[email protected]> wrote:
if (user_mode(regs)) { enter_from_user_mode(); } else { if (!__rcu_is_watching()) { /* * If RCU is not watching then the same careful * sequence vs. lockdep and tracing is required. * * This only happens for IRQs that hit the idle loop, and * even that only happens if we aren't using the sane * MWAIT-while-IF=0 mode. */ lockdep_hardirqs_off(CALLER_ADDR0); rcu_irq_enter(); instrumentation_begin(); trace_hardirqs_off_prepare(); instrumentation_end(); return true; } else { /* * If RCU is watching then the combo function * can be used. */ instrumentation_begin(); trace_hardirqs_off(); rcu_tickle(); instrumentation_end(); } } return false;
Quoted 8 lines
This is exactly what you have except that the
cond_rcupart is gone and I addedrcu_tickle().Paul, the major change here is that if an IRQ hits normal kernel code (i.e. code where RCU is watching and we're not in an EQS), the IRQ won't call
rcu_irq_enter()andrcu_irq_exit(). Instead it will callrcu_tickle()on entry and nothing on exit. Does that cover all the bases?
Just chatted with Paul on IRC and he thinks this should work, but he's not sure whether it's actually sane :)
Thanks,
tglx
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 36 lines · Thomas Gleixner
Quoted 35 lines · Andy Lutomirski
On Wed, May 20, 2020 at 8:36 AM Andy Lutomirski <[email protected]> wrote:
if (user_mode(regs)) { enter_from_user_mode(); } else { if (!__rcu_is_watching()) { /* * If RCU is not watching then the same careful * sequence vs. lockdep and tracing is required. * * This only happens for IRQs that hit the idle loop, and * even that only happens if we aren't using the sane * MWAIT-while-IF=0 mode. */ lockdep_hardirqs_off(CALLER_ADDR0); rcu_irq_enter(); instrumentation_begin(); trace_hardirqs_off_prepare(); instrumentation_end(); return true; } else { /* * If RCU is watching then the combo function * can be used. */ instrumentation_begin(); trace_hardirqs_off(); rcu_tickle(); instrumentation_end(); } } return false;This is exactly what you have except that the
cond_rcupart is gone and I addedrcu_tickle().
Quoted 8 lines
Quoted 5 lines
Paul, the major change here is that if an IRQ hits normal kernel code (i.e. code where RCU is watching and we're not in an EQS), the IRQ won't call
rcu_irq_enter()andrcu_irq_exit(). Instead it will callrcu_tickle()on entry and nothing on exit. Does that cover all the bases?Just chatted with Paul on IRC and he thinks this should work, but he's not sure whether it's actually sane :)
I will have more to say after coding it up. ;-)
Thanx, Paul
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
In reply to Andy LutomirskiQuoted 62 lines · Andy Lutomirski
Quoted 61 lines · Paul E. McKenney
Quoted 60 lines · Andy Lutomirski
Quoted 30 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Quoted 11 lines · Thomas Gleixner
Thomas Gleixner <[email protected]> writes:
It's about this:
rcu_nmi_enter(){ if (!rcu_is_watching()) { make it watch; } else if (!in_nmi()) { do_magic_nohz_dyntick_muck(); }Quoted 8 lines
Quoted 2 lines
So if we do all irq/system vector entries conditional then the
do_magic()gets never executed. After that I got lost...I'm also baffled by that magic, but I'm also not suggesting doing this to *all* entries -- just the not-super-magic ones that use
idtentry_enter().Paul, what is this code actually trying to do?
Citing Paul from IRC:
"The way things are right now, you can leave out the
rcu_irq_enter()
if this is not anohz_fullCPU.Or if this is a
nohz_fullCPU, and the tick is already enabled, in that case you could also leave out thercu_irq_enter().Quoted 8 lines
Or even if this is a
nohz_fullCPU and it does not have the tick enabled, if it has been in the kernel less than a few tens of milliseconds, still OK to avoid invokingrcu_irq_enter()But my guess is that it would be a lot simpler to just always call it.
Hope that helps.
Maybe?
Unless I've missed something, the effect here is that #PF hitting in an RCU-watching context will skip
rcu_irq_enter(), whereas all IRQs (because you converted them) as well as other faults and traps will callrcu_irq_enter().Once upon a time, we did this horrible thing where, on entry from user mode, we would turn on interrupts while still in
CONTEXT_USER, which means we could get an IRQ in an extended quiescent state. This means that the IRQ code had to end the EQS so that IRQ handlers could use RCU. But I killed this a few years ago -- x86 Linux now has a rule that, if IF=1, we are *not* in an EQS with the sole exception of the idle code.In my dream world, we would never ever get IRQs while in an EQS -- we would do MWAIT with IF=0 and we would exit the EQS before taking the interrupt. But I guess we still need to support HLT, which means we have this mess.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
But I still think we can plausibly get rid of the conditional.
You mean the conditional in
rcu_nmi_enter()? In a NO_HZ_FULL=n system, this becomes:So, I meant the conditional in tglx's patch that makes page faults special.
OK.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
If we
get an IRQ or (egads!) a fault in idle context, we'll have !__rcu_is_watching(), but, AFAICT, we also have preemption off.Or we could be early in the kernel-entry code or late in the kernel-exit code, but as far as I know, preemption is disabled on those code paths. As are interrupts, right? And interrupts are disabled on the portions of the CPU-hotplug code where RCU is not watching, if I recall correctly.
Quoted 4 lines
Interrupts are off in the parts of the entry/exit that RCU considers to be user mode. We can get various faults, although these should be either NMI-like or events that genuinely or effectively happened in user mode.
Fair enough!
Quoted 23 lines
Quoted 23 lines
A
nohz_fullCPU does not enable the scheduling-clock interrupt upon entry to the kernel. Normally, this is fine because that CPU will very quickly exit back tonohz_fulluserspace execution, so that RCU will see the quiescent state, either by sampling it directly or by deducing the CPU's passage through that quiescent state by comparing with state that was captured earlier. The grace-period kthread notices the lack of a quiescent state and will eventually set ->rcu_urgent_qs to trigger this code.But if the
nohz_fullCPU stays in the kernel for an extended time, perhaps due to OOM handling or due to processing of some huge I/O that hits in-memory buffers/cache, then RCU needs some way of detecting quiescent states on that CPU. This requires the scheduling-clock interrupt to be alive and well.Are there other ways to get this done? But of course! RCU could for example use
smp_call_function_single()or use workqueues to force execution onto that CPU and enable the tick that way. This gets a little involved in order to avoid deadlock, but if the added check inrcu_nmi_enter()is causing trouble, something can be arranged. Though that something would cause more latency excursions than does the current code.
Quoted 6 lines
Quoted 1 line
Or did you have something else in mind?
I'm trying to understand when we actually need to call the function. Is it just the scheduling interrupt that's supposed to call
rcu_irq_enter()? But the scheduling interrupt is off, so I'm confused.
The scheduling-clock interrupt is indeed off, but if execution remains in the kernel for an extended time period, this becomes a problem. RCU quiescent states don't happen, or if they do, they are not reported to RCU. Grace periods never end, and the system eventually OOMs.
And it is not all that hard to make a CPU stay in the kernel for minutes at a time on a large system.
So what happens is that if RCU notices that a given CPU has not responded in a reasonable time period, it sets that CPU's ->rcu_urgent_qs. This flag plays various roles in various configurations, but on nohz_full CPUs it causes that CPU's next rcu_nmi_enter() invocation to turn that CPU's tick on. It also sets that CPU's ->rcu_forced_tick flag, which prevents redundant turning on of the tick and also causes the quiescent-state detection code to turn off the tick for this CPU.
As you say, the scheduling-clock tick cannot turn itself on, but there might be other interrupts, exceptions, and so on that could. And if nothing like that happens (as might well be the case on a well-isolated CPU), RCU will eventually force one. But it waits a few hundred milliseconds in order to take advantage of whatever naturally occurring interrupt might appear in the meantime.
Does that help?
Thanx, Paul
Andy Lutomirski
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 135 lines · Paul E. McKenney
Quoted 62 lines · Andy Lutomirski
Quoted 61 lines · Paul E. McKenney
Quoted 60 lines · Andy Lutomirski
Quoted 30 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Quoted 11 lines · Thomas Gleixner
Thomas Gleixner <[email protected]> writes:
It's about this:
rcu_nmi_enter(){ if (!rcu_is_watching()) { make it watch; } else if (!in_nmi()) { do_magic_nohz_dyntick_muck(); }Quoted 8 lines
Quoted 2 lines
So if we do all irq/system vector entries conditional then the
do_magic()gets never executed. After that I got lost...I'm also baffled by that magic, but I'm also not suggesting doing this to *all* entries -- just the not-super-magic ones that use
idtentry_enter().Paul, what is this code actually trying to do?
Citing Paul from IRC:
"The way things are right now, you can leave out the
rcu_irq_enter()
if this is not anohz_fullCPU.Or if this is a
nohz_fullCPU, and the tick is already enabled, in that case you could also leave out thercu_irq_enter().Quoted 8 lines
Or even if this is a
nohz_fullCPU and it does not have the tick enabled, if it has been in the kernel less than a few tens of milliseconds, still OK to avoid invokingrcu_irq_enter()But my guess is that it would be a lot simpler to just always call it.
Hope that helps.
Maybe?
Unless I've missed something, the effect here is that #PF hitting in an RCU-watching context will skip
rcu_irq_enter(), whereas all IRQs (because you converted them) as well as other faults and traps will callrcu_irq_enter().Once upon a time, we did this horrible thing where, on entry from user mode, we would turn on interrupts while still in
CONTEXT_USER, which means we could get an IRQ in an extended quiescent state. This means that the IRQ code had to end the EQS so that IRQ handlers could use RCU. But I killed this a few years ago -- x86 Linux now has a rule that, if IF=1, we are *not* in an EQS with the sole exception of the idle code.In my dream world, we would never ever get IRQs while in an EQS -- we would do MWAIT with IF=0 and we would exit the EQS before taking the interrupt. But I guess we still need to support HLT, which means we have this mess.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
But I still think we can plausibly get rid of the conditional.
You mean the conditional in
rcu_nmi_enter()? In a NO_HZ_FULL=n system, this becomes:So, I meant the conditional in tglx's patch that makes page faults special.
OK.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
If we
get an IRQ or (egads!) a fault in idle context, we'll have !__rcu_is_watching(), but, AFAICT, we also have preemption off.Or we could be early in the kernel-entry code or late in the kernel-exit code, but as far as I know, preemption is disabled on those code paths. As are interrupts, right? And interrupts are disabled on the portions of the CPU-hotplug code where RCU is not watching, if I recall correctly.
Quoted 4 lines
Interrupts are off in the parts of the entry/exit that RCU considers to be user mode. We can get various faults, although these should be either NMI-like or events that genuinely or effectively happened in user mode.
Fair enough!
Quoted 23 lines
Quoted 23 lines
A
nohz_fullCPU does not enable the scheduling-clock interrupt upon entry to the kernel. Normally, this is fine because that CPU will very quickly exit back tonohz_fulluserspace execution, so that RCU will see the quiescent state, either by sampling it directly or by deducing the CPU's passage through that quiescent state by comparing with state that was captured earlier. The grace-period kthread notices the lack of a quiescent state and will eventually set ->rcu_urgent_qs to trigger this code.But if the
nohz_fullCPU stays in the kernel for an extended time, perhaps due to OOM handling or due to processing of some huge I/O that hits in-memory buffers/cache, then RCU needs some way of detecting quiescent states on that CPU. This requires the scheduling-clock interrupt to be alive and well.Are there other ways to get this done? But of course! RCU could for example use
smp_call_function_single()or use workqueues to force execution onto that CPU and enable the tick that way. This gets a little involved in order to avoid deadlock, but if the added check inrcu_nmi_enter()is causing trouble, something can be arranged. Though that something would cause more latency excursions than does the current code.Quoted 6 lines
Quoted 1 line
Or did you have something else in mind?
I'm trying to understand when we actually need to call the function. Is it just the scheduling interrupt that's supposed to call
rcu_irq_enter()? But the scheduling interrupt is off, so I'm confused.The scheduling-clock interrupt is indeed off, but if execution remains in the kernel for an extended time period, this becomes a problem. RCU quiescent states don't happen, or if they do, they are not reported to RCU. Grace periods never end, and the system eventually OOMs.
And it is not all that hard to make a CPU stay in the kernel for minutes at a time on a large system.
So what happens is that if RCU notices that a given CPU has not responded in a reasonable time period, it sets that CPU's ->rcu_urgent_qs. This flag plays various roles in various configurations, but on
nohz_fullCPUs it causes that CPU's nextrcu_nmi_enter()invocation to turn that CPU's tick on. It also sets that CPU's ->rcu_forced_tick flag, which prevents redundant turning on of the tick and also causes the quiescent-state detection code to turn off the tick for this CPU.
Quoted 8 lines
As you say, the scheduling-clock tick cannot turn itself on, but there might be other interrupts, exceptions, and so on that could. And if nothing like that happens (as might well be the case on a well-isolated CPU), RCU will eventually force one. But it waits a few hundred milliseconds in order to take advantage of whatever naturally occurring interrupt might appear in the meantime.
Does that help?
Yes, I think. Could this go in a comment in the new function?
Paul E. McKenney
maintainerRe: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
Quoted 143 lines · Andy Lutomirski
Quoted 142 lines · Paul E. McKenney
Quoted 62 lines · Andy Lutomirski
Quoted 61 lines · Paul E. McKenney
Quoted 60 lines · Andy Lutomirski
Quoted 30 lines · Thomas Gleixner
Quoted 12 lines · Andy Lutomirski
Quoted 11 lines · Thomas Gleixner
Thomas Gleixner <[email protected]> writes:
It's about this:
rcu_nmi_enter(){ if (!rcu_is_watching()) { make it watch; } else if (!in_nmi()) { do_magic_nohz_dyntick_muck(); }Quoted 8 lines
Quoted 2 lines
So if we do all irq/system vector entries conditional then the
do_magic()gets never executed. After that I got lost...I'm also baffled by that magic, but I'm also not suggesting doing this to *all* entries -- just the not-super-magic ones that use
idtentry_enter().Paul, what is this code actually trying to do?
Citing Paul from IRC:
"The way things are right now, you can leave out the
rcu_irq_enter()
if this is not anohz_fullCPU.Or if this is a
nohz_fullCPU, and the tick is already enabled, in that case you could also leave out thercu_irq_enter().Quoted 8 lines
Or even if this is a
nohz_fullCPU and it does not have the tick enabled, if it has been in the kernel less than a few tens of milliseconds, still OK to avoid invokingrcu_irq_enter()But my guess is that it would be a lot simpler to just always call it.
Hope that helps.
Maybe?
Unless I've missed something, the effect here is that #PF hitting in an RCU-watching context will skip
rcu_irq_enter(), whereas all IRQs (because you converted them) as well as other faults and traps will callrcu_irq_enter().Once upon a time, we did this horrible thing where, on entry from user mode, we would turn on interrupts while still in
CONTEXT_USER, which means we could get an IRQ in an extended quiescent state. This means that the IRQ code had to end the EQS so that IRQ handlers could use RCU. But I killed this a few years ago -- x86 Linux now has a rule that, if IF=1, we are *not* in an EQS with the sole exception of the idle code.In my dream world, we would never ever get IRQs while in an EQS -- we would do MWAIT with IF=0 and we would exit the EQS before taking the interrupt. But I guess we still need to support HLT, which means we have this mess.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
But I still think we can plausibly get rid of the conditional.
You mean the conditional in
rcu_nmi_enter()? In a NO_HZ_FULL=n system, this becomes:So, I meant the conditional in tglx's patch that makes page faults special.
OK.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
If we
get an IRQ or (egads!) a fault in idle context, we'll have !__rcu_is_watching(), but, AFAICT, we also have preemption off.Or we could be early in the kernel-entry code or late in the kernel-exit code, but as far as I know, preemption is disabled on those code paths. As are interrupts, right? And interrupts are disabled on the portions of the CPU-hotplug code where RCU is not watching, if I recall correctly.
Quoted 4 lines
Interrupts are off in the parts of the entry/exit that RCU considers to be user mode. We can get various faults, although these should be either NMI-like or events that genuinely or effectively happened in user mode.
Fair enough!
Quoted 23 lines
Quoted 23 lines
A
nohz_fullCPU does not enable the scheduling-clock interrupt upon entry to the kernel. Normally, this is fine because that CPU will very quickly exit back tonohz_fulluserspace execution, so that RCU will see the quiescent state, either by sampling it directly or by deducing the CPU's passage through that quiescent state by comparing with state that was captured earlier. The grace-period kthread notices the lack of a quiescent state and will eventually set ->rcu_urgent_qs to trigger this code.But if the
nohz_fullCPU stays in the kernel for an extended time, perhaps due to OOM handling or due to processing of some huge I/O that hits in-memory buffers/cache, then RCU needs some way of detecting quiescent states on that CPU. This requires the scheduling-clock interrupt to be alive and well.Are there other ways to get this done? But of course! RCU could for example use
smp_call_function_single()or use workqueues to force execution onto that CPU and enable the tick that way. This gets a little involved in order to avoid deadlock, but if the added check inrcu_nmi_enter()is causing trouble, something can be arranged. Though that something would cause more latency excursions than does the current code.Quoted 6 lines
Quoted 1 line
Or did you have something else in mind?
I'm trying to understand when we actually need to call the function. Is it just the scheduling interrupt that's supposed to call
rcu_irq_enter()? But the scheduling interrupt is off, so I'm confused.The scheduling-clock interrupt is indeed off, but if execution remains in the kernel for an extended time period, this becomes a problem. RCU quiescent states don't happen, or if they do, they are not reported to RCU. Grace periods never end, and the system eventually OOMs.
And it is not all that hard to make a CPU stay in the kernel for minutes at a time on a large system.
So what happens is that if RCU notices that a given CPU has not responded in a reasonable time period, it sets that CPU's ->rcu_urgent_qs. This flag plays various roles in various configurations, but on
nohz_fullCPUs it causes that CPU's nextrcu_nmi_enter()invocation to turn that CPU's tick on. It also sets that CPU's ->rcu_forced_tick flag, which prevents redundant turning on of the tick and also causes the quiescent-state detection code to turn off the tick for this CPU.As you say, the scheduling-clock tick cannot turn itself on, but there might be other interrupts, exceptions, and so on that could. And if nothing like that happens (as might well be the case on a well-isolated CPU), RCU will eventually force one. But it waits a few hundred milliseconds in order to take advantage of whatever naturally occurring interrupt might appear in the meantime.
Quoted 3 lines
Quoted 1 line
Does that help?
Yes, I think. Could this go in a comment in the new function?
Even if we don't go with the new function, evidence indicates that this commentary should go somewhere. ;-)
Thanx, Paul
Re: [patch V6 12/37] x86/entry: Provide idtentry_entry/exit_cond_rcu()
In reply to Andy LutomirskiQuoted 5 lines · Andy Lutomirski
On Tue, May 19, 2020 at 2:20 PM Thomas Gleixner <[email protected]> wrote:
Unless I've missed something, the effect here is that #PF hitting in an RCU-watching context will skiprcu_irq_enter(), whereas all IRQs (because you converted them) as well as other faults and traps will callrcu_irq_enter().
The only reason why this is needed for #PF is that a kernel mode #PF may sleep. And of course you cannot sleep after calling rcu_irq_enter().
All other interrupts/traps/system vectors cannot sleep ever. So it's a straight forward enter/exit.
Quoted 8 lines
Once upon a time, we did this horrible thing where, on entry from user mode, we would turn on interrupts while still in
CONTEXT_USER, which means we could get an IRQ in an extended quiescent state. This means that the IRQ code had to end the EQS so that IRQ handlers could use RCU. But I killed this a few years ago -- x86 Linux now has a rule that, if IF=1, we are *not* in an EQS with the sole exception of the idle code.
Quoted 4 lines
In my dream world, we would never ever get IRQs while in an EQS -- we would do MWAIT with IF=0 and we would exit the EQS before taking the interrupt. But I guess we still need to support HLT, which means we have this mess.
You always can dream, but dont complain about the nightmares :)
Thanks,
tglx
[tip: x86/entry] x86/entry: Provide idtentry_entry/exit_cond_rcu()
In reply to Thomas GleixnerThe following commit has been merged into the x86/entry branch of tip:
The following commit has been merged into the x86/entry branch of tip:
Commit-ID: 8fbf48a5cdb83a1ae4285920713facef72639641 Gitweb: https://git.kernel.org/tip/8fbf48a5cdb83a1ae4285920713facef72639641 Author: Thomas Gleixner <[email protected]> AuthorDate: Thu, 21 May 2020 22:05:17 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Tue, 26 May 2020 19:06:27 +02:00
x86/entry: Provide idtentry_entry/exit_cond_rcu()
After a lengthy discussion [1] it turned out that RCU does not need a full rcu_irq_enter/exit() when RCU is already watching. All it needs if NOHZ_FULL is active is to check whether the tick needs to be restarted.
This allows to avoid a separate variant for the pagefault handler which cannot invoke rcu_irq_enter() on a kernel pagefault which might sleep.
The cond_rcu argument is only temporary and will be removed once the existing users of idtentry_enter/exit() have been cleaned up. After that the code can be significantly simplified.
[ mingo: Simplified the control flow ]
--- arch/x86/entry/common.c | 79 +++++++++++++++++++++++++------- arch/x86/include/asm/idtentry.h | 14 +++++- 2 files changed, 76 insertions(+), 17 deletions(-)
arch/x86/entry/common.c +64 -15
@@ -512,8 +512,10 @@ SYSCALL_DEFINE0(ni_syscall) }/** -* idtentry_enter - Handle state tracking on idtentry +* idtentry_enter_cond_rcu - Handle state tracking on idtentry with conditional +* RCU handling * @regs: Pointer to pt_regs of interrupted context +* @cond_rcu: Invoke rcu_irq_enter() only if RCU is not watching * * Invokes: * - lockdep irqflag state tracking as low level ASM entry disabled
@@ -521,40 +523,84 @@ SYSCALL_DEFINE0(ni_syscall)* * - Context tracking if the exception hit user mode. * -* - RCU notification if the exception hit kernel mode. -* * - The hardirq tracer to keep the state consistent as low level ASM * entry disabled interrupts. +* +* For kernel mode entries RCU handling is done conditional. If RCU is +* watching then the only RCU requirement is to check whether the tick has +* to be restarted. If RCU is not watching then rcu_irq_enter() has to be +* invoked on entry and rcu_irq_exit() on exit. +* +* Avoiding the rcu_irq_enter/exit() calls is an optimization but also +* solves the problem of kernel mode pagefaults which can schedule, which +* is not possible after invoking rcu_irq_enter() without undoing it. +* +* For user mode entries enter_from_user_mode() must be invoked to +* establish the proper context for NOHZ_FULL. Otherwise scheduling on exit +* would not be possible. +* +* Returns: True if RCU has been adjusted on a kernel entry +* False otherwise +* +* The return value must be fed into the rcu_exit argument of +* idtentry_exit_cond_rcu(). */ -void noinstr idtentry_enter(struct pt_regs *regs) +bool noinstr idtentry_enter_cond_rcu(struct pt_regs *regs, bool cond_rcu) {if (user_mode (regs)) {enter_from_user_mode (); - }else { +return false; + } + +if (!cond_rcu || !__rcu_is_watching ()) { +/* +* If RCU is not watching then the same careful +* sequence vs. lockdep and tracing is required +* as in enter_from_user_mode(). +* +* This only happens for IRQs that hit the idle +* loop, i.e. if idle is not using MWAIT. +*/ lockdep_hardirqs_off (CALLER_ADDR0 );rcu_irq_enter ();instrumentation_begin ();trace_hardirqs_off_prepare ();instrumentation_end (); + +return true; } + +/* +* If RCU is watching then RCU only wants to check +* whether it needs to restart the tick in NOHZ +* mode. +*/ +instrumentation_begin (); +rcu_irq_enter_check_tick (); +/* Use the combo lockdep/tracing function */ +trace_hardirqs_off (); +instrumentation_end (); + +return false; }/** -* idtentry_exit - Common code to handle return from exceptions +* idtentry_exit_cond_rcu - Handle return from exception with conditional RCU +* handling * @regs: Pointer to pt_regs (exception entry regs) +* @rcu_exit: Invoke rcu_irq_exit() if true * * Depending on the return target (kernel/user) this runs the necessary -* preemption and work checks if possible and required and returns to +* preemption and work checks if possible and reguired and returns to * the caller with interrupts disabled and no further work pending. * * This is the last action before returning to the low level ASM code which * just needs to return to the appropriate context. * -* Invoked by all exception/interrupt IDTENTRY handlers which are not -* returning through the paranoid exit path (all except NMI, #DF and the IST -* variants of #MC and #DB) and are therefore on the thread stack. +* Counterpart to idtentry_enter_cond_rcu(). The return value of the entry +* function must be fed into the @rcu_exit argument. */ -void noinstr idtentry_exit(struct pt_regs *regs) +void noinstr idtentry_exit_cond_rcu(struct pt_regs *regs, bool rcu_exit) {lockdep_assert_irqs_disabled ();
@@ -580,7 +626,8 @@ void noinstr idtentry_exit(struct pt_regs *regs)if (IS_ENABLED (CONFIG_DEBUG_ENTRY ))WARN_ON_ONCE (!on_thread_stack ());instrumentation_begin (); -rcu_irq_exit_preempt (); +if (rcu_exit) +rcu_irq_exit_preempt ();if (need_resched ())preempt_schedule_irq ();/* Covers both tracing and lockdep */
@@ -602,10 +649,12 @@ void noinstr idtentry_exit(struct pt_regs *regs)trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (CALLER_ADDR0 );instrumentation_end (); -rcu_irq_exit (); +if (rcu_exit) +rcu_irq_exit ();lockdep_hardirqs_on (CALLER_ADDR0 ); }else { -/* IRQ flags state is correct already. Just tell RCU */ -rcu_irq_exit (); +/* IRQ flags state is correct already. Just tell RCU. */ +if (rcu_exit) +rcu_irq_exit (); } }
arch/x86/include/asm/idtentry.h +12 -2
@@ -7,8 +7,18 @@#ifndef __ASSEMBLY__ -void idtentry_enter (struct pt_regs *regs); -void idtentry_exit (struct pt_regs *regs); +bool idtentry_enter_cond_rcu (struct pt_regs *regs,bool cond_rcu); +void idtentry_exit_cond_rcu (struct pt_regs *regs,bool rcu_exit); + +static __always_inline void idtentry_enter(struct pt_regs *regs) +{ +idtentry_enter_cond_rcu (regs, false); +} + +static __always_inline void idtentry_exit(struct pt_regs *regs) +{ +idtentry_exit_cond_rcu (regs, true); +}/** * DECLARE_IDTENTRY - Declare functions for simple IDT entry points
PATCH v6 13/37 x86/entry: Switch page fault exception to IDTENTRY_RAW
In reply to Thomas GleixnerConvert page fault exceptions to IDTENTRY_RAW:
- Implement the C entry point with DEFINE_IDTENTRY_RAW
- Add the CR2 read into the exception handler
- Add the idtentry_enter/exit_cond_rcu() invocations in in the regular page fault handler and use the regular idtentry_enter/exit() for the async PF part.
- Emit the ASM stub with DECLARE_IDTENTRY_RAW
- Remove the ASM idtentry in 64bit
- Remove the CR2 read from 64bit
- Remove the open coded ASM entry code in 32bit
- Fixup the XEN/PV code
- Remove the old prototypes
No functional change.
arch/x86/entry/entry_32.S +0 -30
@@ -1395,36 +1395,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR, #endif /* CONFIG_HYPERV */ -SYM_CODE_START(page_fault) - ASM_CLAC - pushl $do_page_fault - jmp common_exception_read_cr2 -SYM_CODE_END(page_fault) - -SYM_CODE_START_LOCAL_NOALIGN(common_exception_read_cr2) - /* the function address is in %gs's slot on the stack */ - SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1 - - ENCODE_FRAME_POINTER - - /* fixup %gs */ - GS_TO_REG %ecx - movl PT_GS(%esp), %edi - REG_TO_PTGS %ecx - SET_KERNEL_GS %ecx - - GET_CR2_INTO(%ecx) # might clobber %eax - - /* fixup orig %eax */ - movl PT_ORIG_EAX(%esp), %edx # get the error code - movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart - - TRACE_IRQS_OFF - movl %esp, %eax # pt_regs pointer - CALL_NOSPEC edi - jmp ret_from_exception -SYM_CODE_END(common_exception_read_cr2) - SYM_CODE_START_LOCAL_NOALIGN(common_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -19
@@ -506,15 +506,6 @@ SYM_CODE_END(spurious_entries_start) call error_entry UNWIND_HINT_REGS - .if \vector == X86_TRAP_PF - /* - * Store CR2 early so subsequent faults cannot clobber it. Use R12 as - * intermediate storage as RDX can be clobbered in enter_from_user_mode(). - * GET_CR2_INTO can clobber RAX. - */ - GET_CR2_INTO(%r12); - .endif - .if \sane == 0 TRACE_IRQS_OFF
@@ -533,10 +524,6 @@ SYM_CODE_END(spurious_entries_start) movq $-1, ORIG_RAX(%rsp) /* no syscall to restart */ .endif - .if \vector == X86_TRAP_PF - movq %r12, %rdx /* Move CR2 into 3rd argument */ - .endif - call \cfunc .if \sane == 0
@@ -1060,12 +1047,6 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt #endif /* - * Exception entry points. - */ - -idtentry X86_TRAP_PF page_fault do_page_fault has_error_code=1 - -/* * Reload gs selector with exception handling * edi: new selector *
arch/x86/include/asm/idtentry.h +2 -1
@@ -364,7 +364,8 @@ DECLARE_IDTENTRY_ERRORCODE(X86_TRAP_GP, exc_general_protection);DECLARE_IDTENTRY_ERRORCODE (X86_TRAP_AC , exc_alignment_check);/* Raw exception entries which need extra work */ -DECLARE_IDTENTRY_RAW (X86_TRAP_BP , exc_int3); +DECLARE_IDTENTRY_RAW (X86_TRAP_BP , exc_int3); +DECLARE_IDTENTRY_RAW_ERRORCODE (X86_TRAP_PF , exc_page_fault);#ifdef CONFIG_X86_MCE DECLARE_IDTENTRY_MCE (X86_TRAP_MC , exc_machine_check);
arch/x86/include/asm/traps.h +0 -11
@@ -9,17 +9,6 @@#include <asm/idtentry.h> #include <asm/siginfo.h> /* TRAP_TRACE, ... */ -#define dotraplinkage __visible - -asmlinkage void page_fault(void); -asmlinkage void async_page_fault(void); - -#if defined(CONFIG_X86_64 ) && defined(CONFIG_XEN_PV ) -asmlinkage void xen_page_fault(void); -#endif - -dotraplinkage void do_page_fault(struct pt_regs *regs, unsigned long error_code, unsigned long address); -#ifdef CONFIG_X86_64 asmlinkage __visible notrace struct pt_regs *sync_regs(struct pt_regs *eregs); asmlinkage __visible notrace
arch/x86/kernel/idt.c +2 -2
@@ -59,7 +59,7 @@ static const __initconst struct idt_data early_idts[] = { INTG(X86_TRAP_DB, asm_exc_debug), SYSG(X86_TRAP_BP, asm_exc_int3), #ifdef CONFIG_X86_32 - INTG(X86_TRAP_PF, page_fault), + INTG(X86_TRAP_PF, asm_exc_page_fault), #endif };
@@ -153,7 +153,7 @@ static const __initconst struct idt_data apic_idts[] = {* stacks work only after cpu_init(). */ static const __initconst struct idt_data early_pf_idts[] = { -INTG (X86_TRAP_PF , page_fault), +INTG (X86_TRAP_PF , asm_exc_page_fault), };/*
arch/x86/kernel/kvm.c +8 -6
@@ -218,7 +218,7 @@ void kvm_async_pf_task_wake(u32 token) }EXPORT_SYMBOL_GPL (kvm_async_pf_task_wake); -u32 kvm_read_and_reset_pf_reason (void ) +u32 noinstr kvm_read_and_reset_pf_reason(void) {u32 reason =0 ;
@@ -230,9 +230,8 @@ u32 kvm_read_and_reset_pf_reason(void)return reason; }EXPORT_SYMBOL_GPL (kvm_read_and_reset_pf_reason); -NOKPROBE_SYMBOL (kvm_read_and_reset_pf_reason); -bool __kvm_handle_async_pf (struct pt_regs *regs,u32 token) +noinstr bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token) {u32 reason =kvm_read_and_reset_pf_reason ();
@@ -244,6 +243,9 @@ bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)return false; } +idtentry_enter (regs); +instrumentation_begin (); +/* * If the host managed to inject an async #PF into an interrupt * disabled region, then die hard as this is not going to end well
@@ -258,13 +260,13 @@ bool __kvm_handle_async_pf(struct pt_regs *regs, u32 token)/* Page is swapped out by the host. */ kvm_async_pf_task_wait_schedule (token); }else { -rcu_irq_enter ();kvm_async_pf_task_wake (token); -rcu_irq_exit (); } + +instrumentation_end (); +idtentry_exit (regs);return true; } -NOKPROBE_SYMBOL (__kvm_handle_async_pf);static void __init paravirt_ops_setup (void ) {
arch/x86/mm/fault.c +48 -21
87 lines not on this page · raw message
arch/x86/xen/enlighten_pv.c +1 -1
@@ -627,7 +627,7 @@ static struct trap_array_entry trap_array[] = { #ifdef CONFIG_IA32_EMULATION { entry_INT80_compat, xen_entry_INT80_compat, false }, #endif - { page_fault, xen_page_fault, false }, + TRAP_ENTRY(exc_page_fault, false ), TRAP_ENTRY(exc_divide_error, false ), TRAP_ENTRY(exc_bounds, false ), TRAP_ENTRY(exc_invalid_op, false ),
arch/x86/xen/xen-asm_64.S +1 -1
@@ -43,7 +43,7 @@ xen_pv_trap asm_exc_invalid_tss xen_pv_trap asm_exc_segment_not_present xen_pv_trap asm_exc_stack_segment xen_pv_trap asm_exc_general_protection -xen_pv_trap page_fault +xen_pv_trap asm_exc_page_fault xen_pv_trap asm_exc_spurious_interrupt_bug xen_pv_trap asm_exc_coprocessor_error xen_pv_trap asm_exc_alignment_check
Re: [patch V6 13/37] x86/entry: Switch page fault exception to IDTENTRY_RAW
Quoted 15 lines · Thomas Gleixner
Convert page fault exceptions to
IDTENTRY_RAW:
- Implement the C entry point withDEFINE_IDTENTRY_RAW
- Add the CR2 read into the exception handler
- Add the idtentry_enter/exit_cond_rcu() invocations in in the regular page fault handler and use the regular idtentry_enter/exit() for the async PF part.
- Emit the ASM stub withDECLARE_IDTENTRY_RAW
- Remove the ASM idtentry in 64bit
- Remove the CR2 read from 64bit
- Remove the open coded ASM entry code in 32bit
- Fixup the XEN/PV code
- Remove the old prototypes
although if you make the irq_enter_cond_rcu() mode unconditional, then this comment can go away too:
arch/x86/mm/fault.c do_page_fault(struct pt_regs *regs, unsigned long hw_error_code, · line 1587
Quoted 11 lines
arch/x86/mm/fault.c +11
-0+/* +* Entry handling for valid #PF from kernel mode is slightly +* different: RCU is already watching and rcu_irq_enter() must not +* be invoked because a kernel fault on a user space address might +* sleep. +* +* In case the fault hit a RCU idle region the conditional entry +* code reenabled RCU to avoid subsequent wreckage which helps +* debugability. +*/ + rcu_exit =idtentry_enter_cond_rcu (regs);
--Andy
PATCH v6 14/37 x86/entry: Remove the transition leftovers
In reply to Thomas GleixnerNow that all exceptions are converted over the sane flag is not longer needed. Also the vector argument of idtentry_body on 64 bit is pointless now.
arch/x86/entry/entry_32.S +1 -2
@@ -734,9 +734,8 @@ * @asmsym: ASM symbol for the entry point * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Compatibility flag with 64bit */ -.macro idtentry vector asmsym cfunc has_error_code:req sane=0 +.macro idtentry vector asmsym cfunc has_error_code:req SYM_CODE_START(\asmsym) ASM_CLAC cld
arch/x86/entry/entry_64.S +4 -22
@@ -496,27 +496,14 @@ SYM_CODE_END(spurious_entries_start) /** * idtentry_body - Macro to emit code calling the C function - * @vector: Vector number * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Sane variant which handles irq tracing, context tracking in C */ -.macro idtentry_body vector cfunc has_error_code:req sane=0 +.macro idtentry_body cfunc has_error_code:req call error_entry UNWIND_HINT_REGS - .if \sane == 0 - TRACE_IRQS_OFF - -#ifdef CONFIG_CONTEXT_TRACKING - testb $3, CS(%rsp) - jz .Lfrom_kernel_no_ctxt_tracking_\@ - CALL_enter_from_user_mode -.Lfrom_kernel_no_ctxt_tracking_\@: -#endif - .endif - movq %rsp, %rdi /* pt_regs pointer into 1st argument*/ .if \has_error_code == 1
@@ -526,11 +513,7 @@ SYM_CODE_END(spurious_entries_start) call \cfunc - .if \sane == 0 - jmp error_exit - .else jmp error_return - .endif .endm /**
@@ -539,12 +522,11 @@ SYM_CODE_END(spurious_entries_start) * @asmsym: ASM symbol for the entry point * @cfunc: C function to be called * @has_error_code: Hardware pushed error code on stack - * @sane: Sane variant which handles irq tracing, context tracking in C * * The macro emits code to set up the kernel context for straight forward * and simple IDT entries. No IST stack, no paranoid entry checks. */ -.macro idtentry vector asmsym cfunc has_error_code:req sane=0 +.macro idtentry vector asmsym cfunc has_error_code:req SYM_CODE_START(\asmsym) UNWIND_HINT_IRET_REGS offset=\has_error_code*8 ASM_CLAC
@@ -567,7 +549,7 @@ SYM_CODE_START(\asmsym) .Lfrom_usermode_no_gap_\@: .endif - idtentry_body \vector \cfunc \has_error_code \sane + idtentry_body \cfunc \has_error_code _ASM_NOKPROBE(\asmsym) SYM_CODE_END(\asmsym)
@@ -642,7 +624,7 @@ SYM_CODE_START(\asmsym) /* Switch to the regular task stack and use the noist entry point */ .Lfrom_usermode_switch_stack_\@: - idtentry_body vector noist_\cfunc, has_error_code=0 sane=1 + idtentry_body noist_\cfunc, has_error_code=0 _ASM_NOKPROBE(\asmsym) SYM_CODE_END(\asmsym)
arch/x86/include/asm/idtentry.h +3 -3
@@ -281,10 +281,10 @@ __visible noinstr void func(struct pt_regs *regs, \* The ASM variants for DECLARE_IDTENTRY*() which emit the ASM entry stubs. */ #define DECLARE_IDTENTRY (vector, func) \ - idtentry vector asm_##func func has_error_code=0 sane=1 + idtentry vector asm_##func func has_error_code=0#define DECLARE_IDTENTRY_ERRORCODE (vector, func) \ - idtentry vector asm_##func func has_error_code=1 sane=1 + idtentry vector asm_##func func has_error_code=1/* Special case for 32bit IRET 'trap'. Do not emit ASM code */ #define DECLARE_IDTENTRY_SW (vector, func)
@@ -322,7 +322,7 @@ __visible noinstr void func(struct pt_regs *regs, \/* XEN NMI and DB wrapper */ #define DECLARE_IDTENTRY_XEN (vector, func) \ - idtentry vector asm_exc_xen##func exc_##func has_error_code=0 sane=1 + idtentry vector asm_exc_xen##func exc_##func has_error_code=0#endif /* __ASSEMBLY__ */
Re: [patch V6 14/37] x86/entry: Remove the transition leftovers
Quoted 5 lines · Thomas Gleixner
Now that all exceptions are converted over the sane flag is not longer needed. Also the vector argument of
idtentry_bodyon 64 bit is pointless now.
PATCH v6 15/37 x86/entry: Change exit path of xen_failsafe_callback
In reply to Thomas Gleixnerxen_failsafe_callback is invoked from XEN for two cases:
1. Fault while reloading DS, ES, FS or GS
2. Fault while executing IRET
#1 retries the IRET after XEN has fixed up the segments. #2 injects a #GP which kills the task
For #1 there is no reason to go through the full exception return path because the tasks TIF state is still the same. So just going straight to the IRET path is good enough.
arch/x86/entry/entry_32.S +1 -1
@@ -1352,7 +1352,7 @@ SYM_FUNC_START(xen_failsafe_callback) 5: pushl $-1 /* orig_ax = -1 => not a system call */ SAVE_ALL ENCODE_FRAME_POINTER - jmp ret_from_exception + jmp handle_exception_return .section .fixup, "ax" 6: xorl %eax, %eax
arch/x86/entry/entry_64.S +1 -1
@@ -1175,7 +1175,7 @@ SYM_CODE_START(xen_failsafe_callback) pushq $-1 /* orig_ax = -1 => not a system call */ PUSH_AND_CLEAR_REGS ENCODE_FRAME_POINTER - jmp error_exit + jmp error_return SYM_CODE_END(xen_failsafe_callback) #endif /* CONFIG_XEN_PV */
Re: [patch V6 15/37] x86/entry: Change exit path of xen_failsafe_callback
Quoted 7 lines · Thomas Gleixner
xen_failsafe_callbackis invoked from XEN for two cases:1. Fault while reloading DS, ES, FS or GS
2. Fault while executing IRET
Quoted 6 lines
#1 retries the IRET after XEN has fixed up the segments. #2 injects a #GP which kills the task
For #1 there is no reason to go through the full exception return path because the tasks TIF state is still the same. So just going straight to the IRET path is good enough.
Seems reasonable:
Although a look from a Xen person might be nice too.
PATCH v6 16/37 x86/entry/64: Remove error_exit
In reply to Thomas GleixnerNo more users.
arch/x86/entry/entry_64.S +0 -9
@@ -1356,15 +1356,6 @@ SYM_CODE_START_LOCAL(error_entry) jmp .Lerror_entry_from_usermode_after_swapgs SYM_CODE_END(error_entry) -SYM_CODE_START_LOCAL(error_exit) - UNWIND_HINT_REGS - DISABLE_INTERRUPTS(CLBR_ANY) - TRACE_IRQS_OFF - testb $3, CS(%rsp) - jz retint_kernel - jmp .Lretint_user -SYM_CODE_END(error_exit) - SYM_CODE_START_LOCAL(error_return) UNWIND_HINT_REGS DEBUG_ENTRY_ASSERT_IRQS_OFF
Re: [patch V6 16/37] x86/entry/64: Remove error_exit
Quoted 3 lines · Thomas Gleixner
No more users.
PATCH v6 17/37 x86/entry/32: Remove common_exception
In reply to Thomas GleixnerNo more users.
arch/x86/entry/entry_32.S +0 -21
@@ -1394,27 +1394,6 @@ BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR, #endif /* CONFIG_HYPERV */ -SYM_CODE_START_LOCAL_NOALIGN(common_exception) - /* the function address is in %gs's slot on the stack */ - SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1 - ENCODE_FRAME_POINTER - - /* fixup %gs */ - GS_TO_REG %ecx - movl PT_GS(%esp), %edi # get the function address - REG_TO_PTGS %ecx - SET_KERNEL_GS %ecx - - /* fixup orig %eax */ - movl PT_ORIG_EAX(%esp), %edx # get the error code - movl $-1, PT_ORIG_EAX(%esp) # no syscall to restart - - TRACE_IRQS_OFF - movl %esp, %eax # pt_regs pointer - CALL_NOSPEC edi - jmp ret_from_exception -SYM_CODE_END(common_exception) - SYM_CODE_START_LOCAL_NOALIGN(handle_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
Re: [patch V6 17/37] x86/entry/32: Remove common_exception
Quoted 3 lines · Thomas Gleixner
No more users.
PATCH v6 18/37 x86/irq: Use generic irq_regs implementation
In reply to Thomas GleixnerThe only difference is the name of the per-CPU variable: irq_regs vs. __irq_regs, but the accessor functions are identical.
Remove the pointless copy and use the generic variant.
arch/x86/include/asm/irq_regs.h deleted +0 -32
@@ -1,32 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* -* Per-cpu current frame pointer - the location of the last exception frame on -* the stack, stored in the per-cpu area. -* -* Jeremy Fitzhardinge <[email protected]> -*/ -#ifndef _ASM_X86_IRQ_REGS_H -#define _ASM_X86_IRQ_REGS_H - -#include <asm/percpu.h> - -#define ARCH_HAS_OWN_IRQ_REGS - -DECLARE_PER_CPU(struct pt_regs *, irq_regs); - -static inline struct pt_regs *get_irq_regs (void ) -{ -return __this_cpu_read (irq_regs); -} - -static inline struct pt_regs *set_irq_regs (struct pt_regs *new_regs) -{ -struct pt_regs *old_regs; - - old_regs =get_irq_regs (); -__this_cpu_write (irq_regs, new_regs); - -return old_regs; -} - -#endif /* _ASM_X86_IRQ_REGS_32_H */
arch/x86/kernel/irq.c +0 -3
@@ -26,9 +26,6 @@DEFINE_PER_CPU_SHARED_ALIGNED (irq_cpustat_t, irq_stat);EXPORT_PER_CPU_SYMBOL (irq_stat); -DEFINE_PER_CPU(struct pt_regs *, irq_regs); -EXPORT_PER_CPU_SYMBOL (irq_regs); -atomic_t irq_err_count;/*
PATCH v6 19/37 x86/irq: Convey vector as argument and not in ptregs
In reply to Thomas GleixnerDevice interrupts which go through do_IRQ() or the spurious interrupt handler have their separate entry code on 64 bit for no good reason.
Both 32 and 64 bit transport the vector number through ORIG_[RE]AX in pt_regs. Further the vector number is forced to fit into an u8 and is complemented and offset by 0x80 so it's in the signed character range. Otherwise GAS would expand the pushq to a 5 byte instruction for any vector > 0x7F.
Treat the vector number like an error code and hand it to the C function as argument. This allows to get rid of the extra entry code in a later step.
Simplify the error code push magic by implementing the pushq imm8 via a '.byte 0x6a, vector' sequence so GAS is not able to screw it up. As the pushq imm8 is sign extending the resulting error code needs to be truncated to 8 bits in C code.
arch/x86/entry/calling.h +4 -1
@@ -341,7 +341,10 @@ For 32-bit we have the following conventions - kernel is built with#endif .endm -#endif /* CONFIG_X86_64 */ +#else /* CONFIG_X86_64 */ +# undef UNWIND_HINT_IRET_REGS +# define UNWIND_HINT_IRET_REGS +#endif /* !CONFIG_X86_64 */ .macro STACKLEAK_ERASE#ifdef CONFIG_GCC_PLUGIN_STACKLEAK
arch/x86/entry/entry_32.S +4 -29
@@ -1215,40 +1215,15 @@ SYM_FUNC_END(entry_INT80_32) #endif .endm -/* - * Build the entry stubs with some assembler magic. - * We pack 1 stub into every 8-byte block. - */ - .align 8 -SYM_CODE_START(irq_entries_start) - vector=FIRST_EXTERNAL_VECTOR - .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) - pushl $(~vector+0x80) /* Note: always in signed byte range */ - vector=vector+1 - jmp common_interrupt - .align 8 - .endr -SYM_CODE_END(irq_entries_start) - #ifdef CONFIG_X86_LOCAL_APIC - .align 8 -SYM_CODE_START(spurious_entries_start) - vector=FIRST_SYSTEM_VECTOR - .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) - pushl $(~vector+0x80) /* Note: always in signed byte range */ - vector=vector+1 - jmp common_spurious - .align 8 - .endr -SYM_CODE_END(spurious_entries_start) - SYM_CODE_START_LOCAL(common_spurious) ASM_CLAC - addl $-0x80, (%esp) /* Adjust vector into the [-256, -1] range */ SAVE_ALL switch_stacks=1 ENCODE_FRAME_POINTER TRACE_IRQS_OFF movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ call smp_spurious_interrupt jmp ret_from_intr SYM_CODE_END(common_spurious)
@@ -1261,12 +1236,12 @@ SYM_CODE_END(common_spurious) .p2align CONFIG_X86_L1_CACHE_SHIFT SYM_CODE_START_LOCAL(common_interrupt) ASM_CLAC - addl $-0x80, (%esp) /* Adjust vector into the [-256, -1] range */ - SAVE_ALL switch_stacks=1 ENCODE_FRAME_POINTER TRACE_IRQS_OFF movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ call do_IRQ jmp ret_from_intr SYM_CODE_END(common_interrupt)
arch/x86/entry/entry_64.S +7 -33
@@ -358,34 +358,6 @@ SYM_CODE_START(ret_from_fork) SYM_CODE_END(ret_from_fork) .popsection -/* - * Build the entry stubs with some assembler magic. - * We pack 1 stub into every 8-byte block. - */ - .align 8 -SYM_CODE_START(irq_entries_start) - vector=FIRST_EXTERNAL_VECTOR - .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) - UNWIND_HINT_IRET_REGS - pushq $(~vector+0x80) /* Note: always in signed byte range */ - jmp common_interrupt - .align 8 - vector=vector+1 - .endr -SYM_CODE_END(irq_entries_start) - - .align 8 -SYM_CODE_START(spurious_entries_start) - vector=FIRST_SYSTEM_VECTOR - .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) - UNWIND_HINT_IRET_REGS - pushq $(~vector+0x80) /* Note: always in signed byte range */ - jmp common_spurious - .align 8 - vector=vector+1 - .endr -SYM_CODE_END(spurious_entries_start) - .macro DEBUG_ENTRY_ASSERT_IRQS_OFF #ifdef CONFIG_DEBUG_ENTRY pushq %rax
@@ -755,13 +727,14 @@ _ASM_NOKPROBE(interrupt_entry) /* Interrupt entry/exit. */ /* - * The interrupt stubs push (~vector+0x80) onto the stack and + * The interrupt stubs push vector onto the stack and * then jump to common_spurious/interrupt. */ SYM_CODE_START_LOCAL(common_spurious) - addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1 + movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ + movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ call smp_spurious_interrupt /* rdi points to pt_regs */ jmp ret_from_intr SYM_CODE_END(common_spurious)
@@ -770,10 +743,11 @@ _ASM_NOKPROBE(common_spurious) /* common_interrupt is a hotpath. Align it */ .p2align CONFIG_X86_L1_CACHE_SHIFT SYM_CODE_START_LOCAL(common_interrupt) - addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1 - call do_IRQ /* rdi points to pt_regs */ + movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ + movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ + call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY)
@@ -1022,7 +996,7 @@ apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif apicinterrupt ERROR_APIC_VECTOR error_interrupt smp_error_interrupt -apicinterrupt SPURIOUS_APIC_VECTOR spurious_interrupt smp_spurious_interrupt +apicinterrupt SPURIOUS_APIC_VECTOR spurious_apic_interrupt smp_spurious_apic_interrupt #ifdef CONFIG_IRQ_WORK apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt
arch/x86/include/asm/entry_arch.h +1 -1
@@ -35,7 +35,7 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR) BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR) BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR) -BUILD_INTERRUPT(spurious_interrupt,SPURIOUS_APIC_VECTOR) +BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR) BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR)#ifdef CONFIG_IRQ_WORK
arch/x86/include/asm/hw_irq.h +1 -0
@@ -39,6 +39,7 @@ extern asmlinkage void irq_work_interrupt(void); extern asmlinkage void uv_bau_message_intr1(void); extern asmlinkage void spurious_interrupt(void); +extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +40 -0
@@ -324,6 +324,46 @@ __visible noinstr void func(struct pt_regs *regs, \#define DECLARE_IDTENTRY_XEN (vector, func) \ idtentry vector asm_exc_xen##func exc_##func has_error_code=0 +/* +* ASM code to emit the common vector entry stubs where each stub is +* packed into 8 bytes. +* +* Note, that the 'pushq imm8' is emitted via '.byte 0x6a, vector' because +* GCC treats the local vector variable as unsigned int and would expand +* all vectors above 0x7F to a 5 byte push. The original code did an +* adjustment of the vector number to be in the signed byte range to avoid +* this. While clever it's mindboggling counterintuitive and requires the +* odd conversion back to a real vector number in the C entry points. Using +* .byte achieves the same thing and the only fixup needed in the C entry +* point is to mask off the bits above bit 7 because the push is sign +* extending. +*/ + .align 8 +SYM_CODE_START (irq_entries_start ) + vector=FIRST_EXTERNAL_VECTOR + .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) +UNWIND_HINT_IRET_REGS + .byte 0x6a, vector + jmp common_interrupt + .align 8 + vector=vector+1 + .endr +SYM_CODE_END (irq_entries_start) + +#ifdef CONFIG_X86_LOCAL_APIC + .align 8 +SYM_CODE_START (spurious_entries_start ) + vector=FIRST_SYSTEM_VECTOR + .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) +UNWIND_HINT_IRET_REGS + .byte 0x6a, vector + jmp common_spurious + .align 8 + vector=vector+1 + .endr +SYM_CODE_END(spurious_entries_start) +#endif +#endif /* __ASSEMBLY__ */ /*
arch/x86/include/asm/irq.h +1 -1
@@ -36,7 +36,7 @@ extern void native_init_IRQ(void);extern void handle_irq (struct irq_desc *desc,struct pt_regs *regs); -extern __visible void do_IRQ(struct pt_regs *regs); +extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector);extern void init_ISA_irqs (void );
arch/x86/include/asm/traps.h +2 -1
@@ -41,8 +41,9 @@ asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);#endif void smp_apic_timer_interrupt (struct pt_regs *regs); -void smp_spurious_interrupt (struct pt_regs *regs);void smp_error_interrupt (struct pt_regs *regs); +void smp_spurious_apic_interrupt (struct pt_regs *regs); +void smp_spurious_interrupt (struct pt_regs *regs,unsigned long vector); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +25 -6
@@ -2153,15 +2153,29 @@ void __init register_lapic_address(unsigned long address)* Local APIC interrupts */ -/* -* This interrupt should _never_ happen with our APIC/SMP architecture +/** +* smp_spurious_interrupt - Catch all for interrupts raised on unused vectors +* @regs: Pointer to pt_regs on stack +* @error_code: The vector number is in the lower 8 bits +* +* This is invoked from ASM entry code to catch all interrupts which +* trigger on an entry which is routed to the common_spurious idtentry +* point. +* +* Also called from smp_spurious_apic_interrupt(). */ -__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs) +__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs, +unsigned long vector) { -u8 vector = ~regs->orig_ax;u32 v;entering_irq (); +/* +* The push in the entry ASM code which stores the vector number on +* the stack in the error code slot is sign expanding. Just use the +* lower 8 bits. +*/ + vector &=0xFF ;trace_spurious_apic_entry (vector);inc_irq_stat (irq_spurious_count);
@@ -2182,11 +2196,11 @@ __visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs)*/ v =apic_read (APIC_ISR + ((vector & ~0x1f ) >>1 ));if (v & (1 << (vector &0x1f ))) { -pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n" , +pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n" , vector,smp_processor_id ());ack_APIC_irq (); }else { -pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n" , +pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n" , vector,smp_processor_id ()); } out:
@@ -2194,6 +2208,11 @@ __visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs)exiting_irq (); } +__visible void smp_spurious_apic_interrupt(struct pt_regs *regs) +{ +smp_spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +} +/* * This interrupt should never happen with our APIC/SMP architecture */
arch/x86/kernel/idt.c +1 -1
@@ -142,7 +142,7 @@ static const __initconst struct idt_data apic_idts[] = { #ifdef CONFIG_X86_UV INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), #endif - INTG(SPURIOUS_APIC_VECTOR, spurious_interrupt), + INTG(SPURIOUS_APIC_VECTOR, spurious_apic_interrupt), INTG(ERROR_APIC_VECTOR, error_interrupt), #endif };
arch/x86/kernel/irq.c +9 -5
@@ -227,14 +227,18 @@ u64 arch_irq_stat(void)* SMP cross-CPU interrupts have their own specific * handlers). */ -__visible void __irq_entry do_IRQ(struct pt_regs *regs) +__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) {struct pt_regs *old_regs =set_irq_regs (regs); -struct irq_desc * desc; -/* high bit used in ret_from_ code */ -unsigned vector = ~regs->orig_ax; +struct irq_desc *desc;entering_irq (); +/* +* The push in the entry ASM code which stores the vector number on +* the stack in the error code slot is sign expanding. Just use the +* lower 8 bits. +*/ + vector &=0xFF ;/* entering_irq() tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" );
@@ -249,7 +253,7 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs)ack_APIC_irq ();if (desc ==VECTOR_UNUSED ) { -pr_emerg_ratelimited ("%s: %d.%d No irq handler for vector\n" , +pr_emerg_ratelimited ("%s: %d.%lu No irq handler for vector\n" , __func__,smp_processor_id (), vector); }else {
Re: [patch V6 19/37] x86/irq: Convey vector as argument and not in ptregs
Quoted 11 lines · Thomas Gleixner
Device interrupts which go through
do_IRQ()or the spurious interrupt handler have their separate entry code on 64 bit for no good reason.Both 32 and 64 bit transport the vector number through ORIG_[RE]AX in
pt_regs. Further the vector number is forced to fit into an u8 and is complemented and offset by0x80so it's in the signed character range. Otherwise GAS would expand the pushq to a 5 byte instruction for any vector >0x7F.
Quoted 7 lines
Treat the vector number like an error code and hand it to the C function as argument. This allows to get rid of the extra entry code in a later step.
Simplify the error code push magic by implementing the pushq imm8 via a '.byte
0x6a, vector' sequence so GAS is not able to screw it up. As the pushq imm8 is sign extending the resulting error code needs to be truncated to 8 bits in C code.
although you may be giving me more credit than deserved :)
+ .align 8
arch/x86/include/asm/idtentry.h __visible noinstr void func(struct pt_regs *regs, \ · line 351
Quoted 10 lines
arch/x86/include/asm/idtentry.h +10
-0+SYM_CODE_START (irq_entries_start) + vector=FIRST_EXTERNAL_VECTOR + .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) +UNWIND_HINT_IRET_REGS + .byte 0x6a, vector + jmp common_interrupt + .align 8 + vector=vector+1 + .endr +SYM_CODE_END (irq_entries_start)
Having battled code like this in the past (for early exceptions), I prefer the variant like:
pos = .;
.rept blah blah blah
.byte whatever
jmp whatever
. = pos + 8;
vector = vector + 1
.endr
or maybe:
.rept blah blah blah
.byte whatever
jmp whatever;
. = irq_entries_start + 8 * vector;
vector = vector + 1
.endr
The reason is that these variants will fail to assemble if something goes wrong and the code expands to more than 8 bytes, whereas using .align will cause gas to happily emit 16 bytes and result in hard-to-debug mayhem.
Re: [patch V6 19/37] x86/irq: Convey vector as argument and not in ptregs
Quoted 34 lines · Andy Lutomirski
On Fri, May 15, 2020 at 5:10 PM Thomas Gleixner <[email protected]> wrote:
+ .align 8
arch/x86/include/asm/idtentry.h __visible noinstr void func(struct pt_regs *regs, \ · line 351
Quoted 10 lines
arch/x86/include/asm/idtentry.h +10
-0+SYM_CODE_START (irq_entries_start) + vector=FIRST_EXTERNAL_VECTOR + .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) +UNWIND_HINT_IRET_REGS + .byte 0x6a, vector + jmp common_interrupt + .align 8 + vector=vector+1 + .endr +SYM_CODE_END (irq_entries_start)Having battled code like this in the past (for early exceptions), I prefer the variant like:
pos = .;
.rept blah blah blah
.byte whatever
jmp whatever
. = pos + 8;
vector = vector + 1
.endror maybe:
.rept blah blah blah
.byte whatever
jmp whatever;
. =irq_entries_start+ 8 * vector;
vector = vector + 1
.endr
Quoted 4 lines
The reason is that these variants will fail to assemble if something goes wrong and the code expands to more than 8 bytes, whereas using .align will cause gas to happily emit 16 bytes and result in hard-to-debug mayhem.
Yes. They just make objtool very unhappy:
arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0xfd0: special:
can't find orig instruction
Peter suggested to use:
.pos = .
.byte..
jmp
.nops (pos + 8) - .That works ...
Boris Ostrovsky
reviewerRe: [patch V6 19/37] x86/irq: Convey vector as argument and not in ptregs
Quoted 43 lines · Thomas Gleixner
Quoted 33 lines · Andy Lutomirski
On Fri, May 15, 2020 at 5:10 PM Thomas Gleixner <[email protected]> wrote:
+ .align 8
arch/x86/include/asm/idtentry.h __visible noinstr void func(struct pt_regs *regs, \ · line 351
Quoted 10 lines
arch/x86/include/asm/idtentry.h +10
-0+SYM_CODE_START (irq_entries_start) + vector=FIRST_EXTERNAL_VECTOR + .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) +UNWIND_HINT_IRET_REGS + .byte 0x6a, vector + jmp common_interrupt + .align 8 + vector=vector+1 + .endr +SYM_CODE_END (irq_entries_start)Having battled code like this in the past (for early exceptions), I prefer the variant like:
pos = .;
.rept blah blah blah
.byte whatever
jmp whatever
. = pos + 8;
vector = vector + 1
.endror maybe:
.rept blah blah blah
.byte whatever
jmp whatever;
. =irq_entries_start+ 8 * vector;
vector = vector + 1
.endrQuoted 4 lines
The reason is that these variants will fail to assemble if something goes wrong and the code expands to more than 8 bytes, whereas using .align will cause gas to happily emit 16 bytes and result in hard-to-debug mayhem.
Yes. They just make objtool very unhappy:
arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0xfd0: special:
can't find orig instruction
Quoted 6 lines
Peter suggested to use:
.pos = . .byte.. jmp .nops (pos + 8) - .
Unfortunately this (.nops directive) only works for newer assemblers (2.31, per
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=gas/NEWS;h=9a3f352108e439754688e19e63a6235b38801182;hb=5eb617a71463fa6810cd14f57adfe7a1efc93a96)
I have 2.27 and things don't go well.
-boris
Quoted 2 lines
That works ...
Re: [patch V6 19/37] x86/irq: Convey vector as argument and not in ptregs
Quoted 13 lines · Boris Ostrovsky
Quoted 11 lines
Yes. They just make objtool very unhappy:
arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0xfd0: special:
can't find orig instructionPeter suggested to use:
.pos = . .byte.. jmp .nops (pos + 8) - .
Quoted 6 lines
Unfortunately this (.nops directive) only works for newer assemblers (2.31, per
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=gas/NEWS;h=9a3f352108e439754688e19e63a6235b38801182;hb=5eb617a71463fa6810cd14f57adfe7a1efc93a96)I have 2.27 and things don't go well.
A single nop should be fine, since gas will complain if it tries to move the IP backwards. (Also I'd vote for normal indentation instead of the "assembler magic at 4 spaces" thing.)
.align 8SYM_CODE_START(irq_entries_start)
vector = FIRST_EXTERNAL_VECTOR .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) pos = . UNWIND_HINT_IRET_REGS .byte 0x6a, vector jmp asm_common_interrupt nop . = pos + 8 vector = vector + 1 .endr
SYM_CODE_END(irq_entries_start)
Signature · 1 line
Josh
Re: [patch V6 19/37] x86/irq: Convey vector as argument and not in ptregs
Quoted 19 lines · Josh Poimboeuf
Quoted 18 lines · Boris Ostrovsky
Quoted 11 lines
Yes. They just make objtool very unhappy:
arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0xfd0: special:
can't find orig instructionPeter suggested to use:
.pos = . .byte.. jmp .nops (pos + 8) - .Unfortunately this (.nops directive) only works for newer assemblers (2.31, per
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=gas/NEWS;h=9a3f352108e439754688e19e63a6235b38801182;hb=5eb617a71463fa6810cd14f57adfe7a1efc93a96)
Quoted 4 lines
Quoted 1 line
I have 2.27 and things don't go well.
A single nop should be fine, since gas will complain if it tries to move the IP backwards.
Yes. That's what I posted in the V9 thread :)
Quoted 2 lines
(Also I'd vote for normal indentation instead of the "assembler magic at 4 spaces" thing.)
let me fix that
PATCH v6 20/37 x86/irq/64: Provide handle_irq()
In reply to Thomas GleixnerTo consolidate the interrupt entry/exit code vs. the other exceptions provide handle_irq() (similar to 32bit) to move the interrupt stack switching to C code. That allows to consolidate the entry exit handling by reusing the idtentry machinery both in ASM and C.
arch/x86/kernel/irq_64.c +8 -0
@@ -79,3 +79,11 @@ void do_softirq_own_stack(void)else run_on_irqstack (__do_softirq,NULL ); } + +void handle_irq (struct irq_desc *desc,struct pt_regs *regs) +{ +if (!irq_needs_irq_stack (regs)) +generic_handle_irq_desc (desc); +else +run_on_irqstack (desc->handle_irq, desc); +}
Re: [patch V6 20/37] x86/irq/64: Provide handle_irq()
Quoted 6 lines · Thomas Gleixner
To consolidate the interrupt entry/exit code vs. the other exceptions provide
handle_irq()(similar to 32bit) to move the interrupt stack switching to C code. That allows to consolidate the entry exit handling by reusing the idtentry machinery both in ASM and C.
Quoted 10 lines
Signed-off-by Thomas Gleixner <[email protected]>arch/x86/kernel/irq_64.c +0
-0@@ -79,3 +79,11 @@ void do_softirq_own_stack(void)else
arch/x86/kernel/irq_64.c void do_softirq_own_stack(void) · line 89
Quoted 10 lines
arch/x86/kernel/irq_64.c +8
-0run_on_irqstack (__do_softirq,NULL ); } + +void handle_irq (struct irq_desc *desc,struct pt_regs *regs) +{ +if (!irq_needs_irq_stack (regs)) +generic_handle_irq_desc (desc); +else +run_on_irqstack (desc->handle_irq, desc); +}
Would this be nicer if you open-coded desc->handle_irq(desc) in the if branch to make it look less weird.
This also goes away if you make the run_on_irqstack_if_needed() change I suggested.
PATCH v6 21/37 x86/entry: Add IRQENTRY_IRQ macro
In reply to Thomas GleixnerProvide a seperate IDTENTRY macro for device interrupts. Similar to IDTENTRY_ERRORCODE with the addition of invoking irq_enter/exit_rcu() and providing the errorcode as a 'u8' argument to the C function, which truncates the sign extended vector number.
arch/x86/entry/entry_32.S +14 -0
@@ -751,6 +751,20 @@ SYM_CODE_START(\asmsym) SYM_CODE_END(\asmsym) .endm +.macro idtentry_irq vector cfunc + .p2align CONFIG_X86_L1_CACHE_SHIFT +SYM_CODE_START_LOCAL(asm_\cfunc) + ASM_CLAC + SAVE_ALL switch_stacks=1 + ENCODE_FRAME_POINTER + movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ + movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ + call \cfunc + jmp handle_exception_return +SYM_CODE_END(asm_\cfunc) +.endm + /* * Include the defines which emit the idt entries which are shared * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +14 -0
@@ -528,6 +528,20 @@ SYM_CODE_END(\asmsym) .endm /* + * Interrupt entry/exit. + * + + The interrupt stubs push (vector) onto the stack, which is the error_code + * position of idtentry exceptions, and jump to one of the two idtentry points + * (common/spurious). + * + * common_interrupt is a hotpath, align it to a cache line + */ +.macro idtentry_irq vector cfunc + .p2align CONFIG_X86_L1_CACHE_SHIFT + idtentry \vector asm_\cfunc \cfunc has_error_code=1 +.endm + +/* * MCE and DB exceptions */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +47 -0
59 lines not on this page · raw message
Re: [patch V6 21/37] x86/entry: Add IRQENTRY_IRQ macro
Quoted 6 lines · Thomas Gleixner
Provide a seperate IDTENTRY macro for device interrupts. Similar to
IDTENTRY_ERRORCODEwith the addition of invoking irq_enter/exit_rcu() and providing the errorcode as a 'u8' argument to the C function, which truncates the sign extended vector number.
with a minor minor optimization suggestion:
arch/x86/entry/entry_32.S SYM_CODE_START(\asmsym) · line 761
Quoted 8 lines
+.macro
idtentry_irqvector cfuncarch/x86/entry/entry_32.S +7
-0+ .p2align CONFIG_X86_L1_CACHE_SHIFT +SYM_CODE_START_LOCAL(asm_\cfunc) + ASM_CLAC + SAVE_ALL switch_stacks=1 + ENCODE_FRAME_POINTER + movl %esp, %eax + movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */
You could save somewhere between 0 and 1 cycles by using movzbl here...
Quoted 1 line
+ __##func (regs, (u8)error_code); \
And eliminating this cast. Totally worth it, right?
PATCH v6 22/37 x86/entry: Use idtentry for interrupts
In reply to Thomas GleixnerReplace the extra interrupt handling code and reuse the existing idtentry machinery. This moves the irq stack switching on 64 bit from ASM to C code; 32bit already does the stack switching in C.
arch/x86/entry/entry_32.S +0 -31
@@ -1229,37 +1229,6 @@ SYM_FUNC_END(entry_INT80_32) #endif .endm -#ifdef CONFIG_X86_LOCAL_APIC -SYM_CODE_START_LOCAL(common_spurious) - ASM_CLAC - SAVE_ALL switch_stacks=1 - ENCODE_FRAME_POINTER - TRACE_IRQS_OFF - movl %esp, %eax - movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ - movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ - call smp_spurious_interrupt - jmp ret_from_intr -SYM_CODE_END(common_spurious) -#endif - -/* - * the CPU automatically disables interrupts when executing an IRQ vector, - * so IRQ-flags tracing has to follow that: - */ - .p2align CONFIG_X86_L1_CACHE_SHIFT -SYM_CODE_START_LOCAL(common_interrupt) - ASM_CLAC - SAVE_ALL switch_stacks=1 - ENCODE_FRAME_POINTER - TRACE_IRQS_OFF - movl %esp, %eax - movl PT_ORIG_EAX(%esp), %edx /* get the vector from stack */ - movl $-1, PT_ORIG_EAX(%esp) /* no syscall to restart */ - call do_IRQ - jmp ret_from_intr -SYM_CODE_END(common_interrupt) - #define BUILD_INTERRUPT3(name, nr, fn) \ SYM_FUNC_START(name) \ ASM_CLAC; \
arch/x86/entry/entry_64.S +3 -28
@@ -737,32 +737,7 @@ SYM_CODE_START(interrupt_entry) SYM_CODE_END(interrupt_entry) _ASM_NOKPROBE(interrupt_entry) - -/* Interrupt entry/exit. */ - -/* - * The interrupt stubs push vector onto the stack and - * then jump to common_spurious/interrupt. - */ -SYM_CODE_START_LOCAL(common_spurious) - call interrupt_entry - UNWIND_HINT_REGS indirect=1 - movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ - call smp_spurious_interrupt /* rdi points to pt_regs */ - jmp ret_from_intr -SYM_CODE_END(common_spurious) -_ASM_NOKPROBE(common_spurious) - -/* common_interrupt is a hotpath. Align it */ - .p2align CONFIG_X86_L1_CACHE_SHIFT -SYM_CODE_START_LOCAL(common_interrupt) - call interrupt_entry - UNWIND_HINT_REGS indirect=1 - movq ORIG_RAX(%rdi), %rsi /* get vector from stack */ - movq $-1, ORIG_RAX(%rdi) /* no syscall to restart */ - call do_IRQ /* rdi points to pt_regs */ - /* 0(%rsp): old RSP */ +SYM_CODE_START_LOCAL(common_interrupt_return) ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF
@@ -945,8 +920,8 @@ native_irq_return_ldt: */ jmp native_irq_return_iret #endif -SYM_CODE_END(common_interrupt) -_ASM_NOKPROBE(common_interrupt) +SYM_CODE_END(common_interrupt_return) +_ASM_NOKPROBE(common_interrupt_return) /* * APIC interrupts.
arch/x86/include/asm/hw_irq.h +0 -1
@@ -38,7 +38,6 @@ extern asmlinkage void error_interrupt(void); extern asmlinkage void irq_work_interrupt(void); extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void spurious_interrupt(void); extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +8 -2
@@ -391,7 +391,7 @@ SYM_CODE_START(irq_entries_start) .rept (FIRST_SYSTEM_VECTOR - FIRST_EXTERNAL_VECTOR) UNWIND_HINT_IRET_REGS .byte 0x6a, vector - jmp common_interrupt + jmp asm_common_interrupt .align 8 vector=vector+1 .endr
@@ -404,7 +404,7 @@ SYM_CODE_START(spurious_entries_start) .rept (NR_VECTORS - FIRST_SYSTEM_VECTOR) UNWIND_HINT_IRET_REGS .byte 0x6a, vector - jmp common_spurious + jmp asm_spurious_interrupt .align 8 vector=vector+1 .endr
@@ -475,6 +475,12 @@ DECLARE_IDTENTRY_DF(X86_TRAP_DF, exc_double_fault);DECLARE_IDTENTRY (X86_TRAP_OTHER , exc_xen_hypervisor_callback);#endif +/* Device interrupts common/spurious */ +DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , common_interrupt); +#ifdef CONFIG_X86_LOCAL_APIC +DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , spurious_interrupt); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq.h +0 -2
@@ -36,8 +36,6 @@ extern void native_init_IRQ(void);extern void handle_irq (struct irq_desc *desc,struct pt_regs *regs); -extern __visible void do_IRQ(struct pt_regs *regs, unsigned long vector); -extern void init_ISA_irqs (void );extern void __init init_IRQ (void );
arch/x86/include/asm/traps.h +0 -1
@@ -43,7 +43,6 @@ asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);void smp_apic_timer_interrupt (struct pt_regs *regs);void smp_error_interrupt (struct pt_regs *regs);void smp_spurious_apic_interrupt (struct pt_regs *regs); -void smp_spurious_interrupt (struct pt_regs *regs,unsigned long vector); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +8 -15
@@ -2154,9 +2154,9 @@ void __init register_lapic_address(unsigned long address)*/ /** -* smp_spurious_interrupt - Catch all for interrupts raised on unused vectors +* spurious_interrupt - Catch all for interrupts raised on unused vectors * @regs: Pointer to pt_regs on stack -* @error_code: The vector number is in the lower 8 bits +* @vector: The vector number * * This is invoked from ASM entry code to catch all interrupts which * trigger on an entry which is routed to the common_spurious idtentry
@@ -2164,18 +2164,10 @@ void __init register_lapic_address(unsigned long address)* * Also called from smp_spurious_apic_interrupt(). */ -__visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs, -unsigned long vector) +DEFINE_IDTENTRY_IRQ (spurious_interrupt) {u32 v; -entering_irq (); -/* -* The push in the entry ASM code which stores the vector number on -* the stack in the error code slot is sign expanding. Just use the -* lower 8 bits. -*/ - vector &=0xFF ;trace_spurious_apic_entry (vector);inc_irq_stat (irq_spurious_count);
@@ -2196,21 +2188,22 @@ __visible void __irq_entry smp_spurious_interrupt(struct pt_regs *regs,*/ v =apic_read (APIC_ISR + ((vector & ~0x1f ) >>1 ));if (v & (1 << (vector &0x1f ))) { -pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Acked\n" , +pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Acked\n" , vector,smp_processor_id ());ack_APIC_irq (); }else { -pr_info ("Spurious interrupt (vector 0x%02lx) on CPU#%d. Not pending!\n" , +pr_info ("Spurious interrupt (vector 0x%02x) on CPU#%d. Not pending!\n" , vector,smp_processor_id ()); } out:trace_spurious_apic_exit (vector); -exiting_irq (); } __visible void smp_spurious_apic_interrupt(struct pt_regs *regs) { -smp_spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +entering_irq (); +__spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); +exiting_irq (); }/*
arch/x86/kernel/apic/msi.c +2 -1
@@ -115,7 +115,8 @@ msi_set_affinity(struct irq_data *irqd, const struct cpumask *mask, bool force)* denote it as spurious which is no harm as this is a rare event * and interrupt handlers have to cope with spurious interrupts * anyway. If the vector is unused, then it is marked so it won't -* trigger the 'No irq handler for vector' warning in do_IRQ(). +* trigger the 'No irq handler for vector' warning in +* common_interrupt(). * * This requires to hold vector lock to prevent concurrent updates to * the affected vector.
arch/x86/kernel/irq.c +7 -20
@@ -19,6 +19,7 @@#include <asm/mce.h> #include <asm/hw_irq.h> #include <asm/desc.h> +#include <asm/traps.h> #define CREATE_TRACE_POINTS #include <asm/trace/irq_vectors.h>
@@ -223,37 +224,25 @@ u64 arch_irq_stat(void)/* -* do_IRQ handles all normal device IRQ's (the special -* SMP cross-CPU interrupts have their own specific -* handlers). +* common_interrupt() handles all normal device IRQ's (the special SMP +* cross-CPU interrupts have their own entry points). */ -__visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) +DEFINE_IDTENTRY_IRQ (common_interrupt) {struct pt_regs *old_regs =set_irq_regs (regs);struct irq_desc *desc; -entering_irq (); -/* -* The push in the entry ASM code which stores the vector number on -* the stack in the error code slot is sign expanding. Just use the -* lower 8 bits. -*/ - vector &=0xFF ; - -/* entering_irq() tells RCU that we're not quiescent. Check it. */ +/* entry code tells RCU that we're not quiescent. Check it. */ RCU_LOCKDEP_WARN (!rcu_is_watching (),"IRQ failed to wake up RCU" ); desc =__this_cpu_read (vector_irq[vector]);if (likely (!IS_ERR_OR_NULL (desc))) { -if (IS_ENABLED (CONFIG_X86_32 )) -handle_irq (desc, regs); -else -generic_handle_irq_desc (desc); +handle_irq (desc, regs); }else {ack_APIC_irq ();if (desc ==VECTOR_UNUSED ) { -pr_emerg_ratelimited ("%s: %d.%lu No irq handler for vector\n" , +pr_emerg_ratelimited ("%s: %d.%u No irq handler for vector\n" , __func__,smp_processor_id (), vector); }else {
@@ -261,8 +250,6 @@ __visible void __irq_entry do_IRQ(struct pt_regs *regs, unsigned long vector) } } -exiting_irq (); -set_irq_regs (old_regs); }
Re: [patch V6 22/37] x86/entry: Use idtentry for interrupts
Quoted 5 lines · Thomas Gleixner
Replace the extra interrupt handling code and reuse the existing idtentry machinery. This moves the irq stack switching on 64 bit from ASM to C code; 32bit already does the stack switching in C.
PATCH v6 23/37 x86/entry: Provide IDTENTRY_SYSVEC
In reply to Thomas GleixnerProvide a IDTENTRY variant for system vectors to consolidate the different mechanisms to emit the ASM stubs for 32 an 64 bit.
On 64bit this also moves the stack switching from ASM to C code. 32bit will excute the system vectors w/o stack switching as before. As some of the system vector handlers require access to pt_regs this requires a new stack switching macro which can handle an argument.
The alternative solution would be to implement the set_irq_regs() dance right in the entry macro, but most system vector handlers do not require it, so avoid the overhead.
Provide the entry/exit handling as inline functions so the scheduler IPI can use it to implement lightweight entry handling depending on trace point enablement. This ensures that the code is consistent.
arch/x86/entry/entry_32.S +4 -0
@@ -765,6 +765,10 @@ SYM_CODE_START_LOCAL(asm_\cfunc) SYM_CODE_END(asm_\cfunc) .endm +.macro idtentry_sysvec vector cfunc + idtentry \vector asm_\cfunc \cfunc has_error_code=0 +.endm + /* * Include the defines which emit the idt entries which are shared * shared between 32 and 64 bit.
arch/x86/entry/entry_64.S +8 -0
@@ -542,6 +542,14 @@ SYM_CODE_END(\asmsym) .endm /* + * System vectors which invoke their handlers directly and are not + * going through the regular common device interrupt handling code. + */ +.macro idtentry_sysvec vector cfunc + idtentry \vector asm_\cfunc \cfunc has_error_code=0 +.endm + +/* * MCE and DB exceptions */ #define CPU_TSS_IST(x) PER_CPU_VAR(cpu_tss_rw) + (TSS_ist + (x) * 8)
arch/x86/include/asm/idtentry.h +86 -0
104 lines not on this page · raw message
Re: [patch V6 23/37] x86/entry: Provide IDTENTRY_SYSVEC
Quoted 9 lines · Thomas Gleixner
Provide a IDTENTRY variant for system vectors to consolidate the different mechanisms to emit the ASM stubs for 32 an 64 bit.
On 64bit this also moves the stack switching from ASM to C code. 32bit will excute the system vectors w/o stack switching as before. As some of the system vector handlers require access to
pt_regsthis requires a new stack switching macro which can handle an argument.
Is that last sentence obsolete?
Otherwise,
Re: [patch V6 23/37] x86/entry: Provide IDTENTRY_SYSVEC
Quoted 6 lines · Andy Lutomirski
Quoted 5 lines · Thomas Gleixner
Provide a IDTENTRY variant for system vectors to consolidate the different mechanisms to emit the ASM stubs for 32 an 64 bit.
Quoted 6 lines
Quoted 4 lines
On 64bit this also moves the stack switching from ASM to C code. 32bit will excute the system vectors w/o stack switching as before. As some of the system vector handlers require access to
pt_regsthis requires a new stack switching macro which can handle an argument.Is that last sentence obsolete?
Yes.
Quoted 3 lines
Otherwise,
Acked-by Andy Lutomirski <[email protected]>
PATCH v6 24/37 x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC
In reply to Thomas GleixnerConvert APIC interrupts to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
No functional change.
arch/x86/entry/entry_64.S +0 -6
@@ -965,9 +965,6 @@ apicinterrupt3 REBOOT_VECTOR reboot_interrupt smp_reboot_interrupt apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt #endif -apicinterrupt LOCAL_TIMER_VECTOR apic_timer_interrupt smp_apic_timer_interrupt -apicinterrupt X86_PLATFORM_IPI_VECTOR x86_platform_ipi smp_x86_platform_ipi - #ifdef CONFIG_HAVE_KVM apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_posted_intr_wakeup_ipi
@@ -992,9 +989,6 @@ apicinterrupt CALL_FUNCTION_VECTOR call_function_interrupt smp_call_function_i apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif -apicinterrupt ERROR_APIC_VECTOR error_interrupt smp_error_interrupt -apicinterrupt SPURIOUS_APIC_VECTOR spurious_apic_interrupt smp_spurious_apic_interrupt - #ifdef CONFIG_IRQ_WORK apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -5
@@ -33,11 +33,6 @@ BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)*/ #ifdef CONFIG_X86_LOCAL_APIC -BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR) -BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR) -BUILD_INTERRUPT(spurious_apic_interrupt,SPURIOUS_APIC_VECTOR) -BUILD_INTERRUPT(x86_platform_ipi, X86_PLATFORM_IPI_VECTOR) -#ifdef CONFIG_IRQ_WORK BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR)#endif
arch/x86/include/asm/hw_irq.h +0 -4
@@ -29,16 +29,12 @@#include <asm/sections.h> /* Interrupt handlers registered during init_IRQ */ -extern asmlinkage void apic_timer_interrupt(void); -extern asmlinkage void x86_platform_ipi(void); extern asmlinkage void kvm_posted_intr_ipi(void); extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); extern asmlinkage void kvm_posted_intr_nested_ipi(void); -extern asmlinkage void error_interrupt(void); extern asmlinkage void irq_work_interrupt(void); extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void spurious_apic_interrupt(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void);
arch/x86/include/asm/idtentry.h +8 -0
@@ -567,6 +567,14 @@ DECLARE_IDTENTRY_IRQ(X86_TRAP_OTHER, common_interrupt);DECLARE_IDTENTRY_IRQ (X86_TRAP_OTHER , spurious_interrupt);#endif +/* System vector entry points */ +#ifdef CONFIG_X86_LOCAL_APIC +DECLARE_IDTENTRY_SYSVEC (ERROR_APIC_VECTOR , sysvec_error_interrupt); +DECLARE_IDTENTRY_SYSVEC (SPURIOUS_APIC_VECTOR , sysvec_spurious_apic_interrupt); +DECLARE_IDTENTRY_SYSVEC (LOCAL_TIMER_VECTOR , sysvec_apic_timer_interrupt); +DECLARE_IDTENTRY_SYSVEC (X86_PLATFORM_IPI_VECTOR , sysvec_x86_platform_ipi); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq.h +0 -1
@@ -44,7 +44,6 @@ extern void __init init_IRQ(void);void arch_trigger_cpumask_backtrace (const struct cpumask *mask,bool exclude_self); -extern __visible void smp_x86_platform_ipi(struct pt_regs *regs);#define arch_trigger_cpumask_backtrace arch_trigger_cpumask_backtrace#endif
arch/x86/include/asm/traps.h +0 -3
@@ -40,9 +40,6 @@ asmlinkage void smp_threshold_interrupt(struct pt_regs *regs); asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);#endif -void smp_apic_timer_interrupt (struct pt_regs *regs); -void smp_error_interrupt (struct pt_regs *regs); -void smp_spurious_apic_interrupt (struct pt_regs *regs); asmlinkage void smp_irq_move_cleanup_interrupt(void);#ifdef CONFIG_VMAP_STACK
arch/x86/kernel/apic/apic.c +5 -18
@@ -1121,23 +1121,14 @@ static void local_apic_timer_interrupt(void)* [ if a single-CPU system runs an SMP kernel then we call the local * interrupt as well. Thus we cannot inline the local irq ... ] */ -__visible void __irq_entry smp_apic_timer_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_apic_timer_interrupt) {struct pt_regs *old_regs =set_irq_regs (regs); -/* -* NOTE! We'd better ACK the irq immediately, -* because timer handling can be slow. -* -* update_process_times() expects us to have done irq_enter(). -* Besides, if we don't timer interrupts ignore the global -* interrupt lock, which is the WrongThing (tm) to do. -*/ -entering_ack_irq (); +ack_APIC_irq ();trace_local_timer_entry (LOCAL_TIMER_VECTOR );local_apic_timer_interrupt ();trace_local_timer_exit (LOCAL_TIMER_VECTOR ); -exiting_irq ();set_irq_regs (old_regs); }
@@ -2162,7 +2153,7 @@ void __init register_lapic_address(unsigned long address)* trigger on an entry which is routed to the common_spurious idtentry * point. * -* Also called from smp_spurious_apic_interrupt(). +* Also called from sysvec_spurious_apic_interrupt(). */ DEFINE_IDTENTRY_IRQ (spurious_interrupt) {
@@ -2199,17 +2190,15 @@ DEFINE_IDTENTRY_IRQ(spurious_interrupt)trace_spurious_apic_exit (vector); } -__visible void smp_spurious_apic_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_spurious_apic_interrupt) { -entering_irq ();__spurious_interrupt (regs,SPURIOUS_APIC_VECTOR ); -exiting_irq (); }/* * This interrupt should never happen with our APIC/SMP architecture */ -__visible void __irq_entry smp_error_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_error_interrupt) {static const char *const error_interrupt_reason[] = {"Send CS error" ,/* APIC Error Bit 0 */
@@ -2223,7 +2212,6 @@ __visible void __irq_entry smp_error_interrupt(struct pt_regs *regs) };u32 v, i =0 ; -entering_irq ();trace_error_apic_entry (ERROR_APIC_VECTOR );/* First tickle the hardware, only then report what went on. -- REW */
@@ -2247,7 +2235,6 @@ __visible void __irq_entry smp_error_interrupt(struct pt_regs *regs)apic_printk (APIC_DEBUG ,KERN_CONT "\n" );trace_error_apic_exit (ERROR_APIC_VECTOR ); -exiting_irq (); }/**
arch/x86/kernel/idt.c +4 -4
@@ -129,8 +129,8 @@ static const __initconst struct idt_data apic_idts[] = {#endif #ifdef CONFIG_X86_LOCAL_APIC - INTG(LOCAL_TIMER_VECTOR, apic_timer_interrupt), - INTG(X86_PLATFORM_IPI_VECTOR, x86_platform_ipi), + INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), + INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi),# ifdef CONFIG_HAVE_KVM INTG (POSTED_INTR_VECTOR , kvm_posted_intr_ipi), INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi),
@@ -142,8 +142,8 @@ static const __initconst struct idt_data apic_idts[] = { #ifdef CONFIG_X86_UV INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), #endif - INTG(SPURIOUS_APIC_VECTOR, spurious_apic_interrupt), - INTG(ERROR_APIC_VECTOR, error_interrupt), + INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), + INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), #endif };
arch/x86/kernel/irq.c +3 -3
@@ -14,6 +14,7 @@#include <linux/irq.h> #include <asm/apic.h> +#include <asm/traps.h> #include <asm/io_apic.h> #include <asm/irq.h> #include <asm/mce.h>
@@ -259,17 +260,16 @@ void (*x86_platform_ipi_callback)(void) = NULL;/* * Handler for X86_PLATFORM_IPI_VECTOR. */ -__visible void __irq_entry smp_x86_platform_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_x86_platform_ipi) {struct pt_regs *old_regs =set_irq_regs (regs); -entering_ack_irq (); +ack_APIC_irq ();trace_x86_platform_ipi_entry (X86_PLATFORM_IPI_VECTOR );inc_irq_stat (x86_platform_ipis);if (x86_platform_ipi_callback)x86_platform_ipi_callback ();trace_x86_platform_ipi_exit (X86_PLATFORM_IPI_VECTOR ); -exiting_irq ();set_irq_regs (old_regs); }#endif
Re: [patch V6 24/37] x86/entry: Convert APIC interrupts to IDTENTRY_SYSVEC
Quoted 11 lines · Thomas Gleixner
Convert APIC interrupts to
IDTENTRY_SYSVEC
- Implement the C entry point withDEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub withDECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove theBUILD_INTERRUPTentries in 32bit
- Remove the old prototypesNo functional change.
PATCH v6 25/37 x86/entry: Convert SMP system vectors to IDTENTRY_SYSVEC
In reply to Thomas GleixnerConvert SMP system vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
No functional change.
arch/x86/entry/entry_64.S +0 -7
@@ -956,11 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm -#ifdef CONFIG_SMP -apicinterrupt3 IRQ_MOVE_CLEANUP_VECTOR irq_move_cleanup_interrupt smp_irq_move_cleanup_interrupt -apicinterrupt3 REBOOT_VECTOR reboot_interrupt smp_reboot_interrupt -#endif - #ifdef CONFIG_X86_UV apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt #endif
@@ -984,8 +979,6 @@ apicinterrupt THERMAL_APIC_VECTOR thermal_interrupt smp_thermal_interrupt #endif #ifdef CONFIG_SMP -apicinterrupt CALL_FUNCTION_SINGLE_VECTOR call_function_single_interrupt smp_call_function_single_interrupt -apicinterrupt CALL_FUNCTION_VECTOR call_function_interrupt smp_call_function_interrupt apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -4
@@ -12,10 +12,6 @@*/ #ifdef CONFIG_SMP BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR) -BUILD_INTERRUPT(call_function_interrupt,CALL_FUNCTION_VECTOR) -BUILD_INTERRUPT(call_function_single_interrupt,CALL_FUNCTION_SINGLE_VECTOR) -BUILD_INTERRUPT(irq_move_cleanup_interrupt, IRQ_MOVE_CLEANUP_VECTOR) -BUILD_INTERRUPT(reboot_interrupt, REBOOT_VECTOR)#endif #ifdef CONFIG_HAVE_KVM
arch/x86/include/asm/hw_irq.h +0 -5
@@ -38,14 +38,9 @@ extern asmlinkage void uv_bau_message_intr1(void); extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void); -extern asmlinkage void irq_move_cleanup_interrupt(void); -extern asmlinkage void reboot_interrupt(void); extern asmlinkage void threshold_interrupt(void); extern asmlinkage void deferred_error_interrupt(void); -extern asmlinkage void call_function_interrupt(void); -extern asmlinkage void call_function_single_interrupt(void); -#ifdef CONFIG_X86_LOCAL_APIC struct irq_data ;struct pci_dev ;
arch/x86/include/asm/idtentry.h +7 -0
@@ -575,6 +575,13 @@ DECLARE_IDTENTRY_SYSVEC(LOCAL_TIMER_VECTOR, sysvec_apic_timer_interrupt);DECLARE_IDTENTRY_SYSVEC (X86_PLATFORM_IPI_VECTOR , sysvec_x86_platform_ipi);#endif +#ifdef CONFIG_SMP +DECLARE_IDTENTRY_SYSVEC (IRQ_MOVE_CLEANUP_VECTOR , sysvec_irq_move_cleanup); +DECLARE_IDTENTRY_SYSVEC (REBOOT_VECTOR , sysvec_reboot); +DECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_SINGLE_VECTOR , sysvec_call_function_single); +DECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_VECTOR , sysvec_call_function); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/traps.h +0 -2
@@ -40,8 +40,6 @@ asmlinkage void smp_threshold_interrupt(struct pt_regs *regs); asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs);#endif -asmlinkage void smp_irq_move_cleanup_interrupt(void); -#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message, struct pt_regs *regs,
arch/x86/kernel/apic/vector.c +2 -3
@@ -861,13 +861,13 @@ static void free_moved_vector(struct apic_chip_data *apicd) apicd->move_in_progress =0 ; } -asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void) +DEFINE_IDTENTRY_SYSVEC (sysvec_irq_move_cleanup) {struct hlist_head *clhead =this_cpu_ptr (&cleanup_list);struct apic_chip_data *apicd;struct hlist_node *tmp; -entering_ack_irq (); +ack_APIC_irq ();/* Prevent vectors vanishing under us */ raw_spin_lock (&vector_lock);
@@ -892,7 +892,6 @@ asmlinkage __visible void __irq_entry smp_irq_move_cleanup_interrupt(void) }raw_spin_unlock (&vector_lock); -exiting_irq (); }static void __send_cleanup_vector (struct apic_chip_data *apicd)
arch/x86/kernel/idt.c +5 -5
@@ -109,11 +109,11 @@ static const __initconst struct idt_data def_idts[] = {*/ static const __initconst struct idt_data apic_idts[] = { #ifdef CONFIG_SMP -INTG (RESCHEDULE_VECTOR , reschedule_interrupt), -INTG (CALL_FUNCTION_VECTOR , call_function_interrupt), -INTG (CALL_FUNCTION_SINGLE_VECTOR , call_function_single_interrupt), -INTG (IRQ_MOVE_CLEANUP_VECTOR , irq_move_cleanup_interrupt), -INTG (REBOOT_VECTOR , reboot_interrupt), +INTG (RESCHEDULE_VECTOR , reschedule_interrupt), +INTG (CALL_FUNCTION_VECTOR , asm_sysvec_call_function), +INTG (CALL_FUNCTION_SINGLE_VECTOR , asm_sysvec_call_function_single), +INTG (IRQ_MOVE_CLEANUP_VECTOR , asm_sysvec_irq_move_cleanup), +INTG (REBOOT_VECTOR , asm_sysvec_reboot), #endif #ifdef CONFIG_X86_THERMAL_VECTOR
arch/x86/kernel/smp.c +7 -11
@@ -27,6 +27,7 @@#include <asm/mmu_context.h> #include <asm/proto.h> #include <asm/apic.h> +#include <asm/idtentry.h> #include <asm/nmi.h> #include <asm/mce.h> #include <asm/trace/irq_vectors.h>
@@ -130,13 +131,11 @@ static int smp_stop_nmi_callback(unsigned int val, struct pt_regs *regs)/* * this function calls the 'stop' function on all other CPUs in the system. */ - -asmlinkage __visible void smp_reboot_interrupt(void) +DEFINE_IDTENTRY_SYSVEC (sysvec_reboot) { -ipi_entering_ack_irq (); +ack_APIC_irq ();cpu_emergency_vmxoff ();stop_this_cpu (NULL ); -irq_exit (); }static int register_stop_handler (void )
@@ -227,7 +226,6 @@ __visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs) {ack_APIC_irq ();inc_irq_stat (irq_resched_count); -kvm_set_cpu_l1tf_flush_l1d ();if (trace_resched_ipi_enabled ()) {/*
@@ -244,24 +242,22 @@ __visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs)scheduler_ipi (); } -__visible void __irq_entry smp_call_function_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_call_function) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_call_function_entry (CALL_FUNCTION_VECTOR );inc_irq_stat (irq_call_count);generic_smp_call_function_interrupt ();trace_call_function_exit (CALL_FUNCTION_VECTOR ); -exiting_irq (); } -__visible void __irq_entry smp_call_function_single_interrupt(struct pt_regs *r) +DEFINE_IDTENTRY_SYSVEC (sysvec_call_function_single) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_call_function_single_entry (CALL_FUNCTION_SINGLE_VECTOR );inc_irq_stat (irq_call_count);generic_smp_call_function_single_interrupt ();trace_call_function_single_exit (CALL_FUNCTION_SINGLE_VECTOR ); -exiting_irq (); }static int __init nonmi_ipi_setup (char *str)
Re: [patch V6 25/37] x86/entry: Convert SMP system vectors to IDTENTRY_SYSVEC
Quoted 8 lines · Thomas Gleixner
Convert SMP system vectors to
IDTENTRY_SYSVEC
- Implement the C entry point withDEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub withDECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove theBUILD_INTERRUPTentries in 32bit
- Remove the old prototypes
PATCH v6 26/37 x86/entry: Convert various system vectors
In reply to Thomas GleixnerConvert various system vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
No functional change.
arch/x86/entry/entry_64.S +0 -19
@@ -956,9 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm -#ifdef CONFIG_X86_UV -apicinterrupt3 UV_BAU_MESSAGE uv_bau_message_intr1 uv_bau_message_interrupt -#endif #ifdef CONFIG_HAVE_KVM apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi
@@ -966,26 +963,10 @@ apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_post apicinterrupt3 POSTED_INTR_NESTED_VECTOR kvm_posted_intr_nested_ipi smp_kvm_posted_intr_nested_ipi #endif -#ifdef CONFIG_X86_MCE_THRESHOLD -apicinterrupt THRESHOLD_APIC_VECTOR threshold_interrupt smp_threshold_interrupt -#endif - -#ifdef CONFIG_X86_MCE_AMD -apicinterrupt DEFERRED_ERROR_VECTOR deferred_error_interrupt smp_deferred_error_interrupt -#endif - -#ifdef CONFIG_X86_THERMAL_VECTOR -apicinterrupt THERMAL_APIC_VECTOR thermal_interrupt smp_thermal_interrupt -#endif - #ifdef CONFIG_SMP apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif -#ifdef CONFIG_IRQ_WORK -apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt -#endif - /* * Reload gs selector with exception handling * edi: new selector
arch/x86/include/asm/apic.h +0 -13
@@ -534,24 +534,11 @@ static inline void entering_ack_irq(void) ack_APIC_irq(); } -static inline void ipi_entering_ack_irq (void ) -{ -irq_enter (); -ack_APIC_irq (); -kvm_set_cpu_l1tf_flush_l1d (); -} -static inline void exiting_irq (void ) {irq_exit (); } -static inline void exiting_ack_irq (void ) -{ -ack_APIC_irq (); -irq_exit (); -} -extern void ioapic_zap_locks (void );#endif /* _ASM_X86_APIC_H */
arch/x86/include/asm/entry_arch.h +0 -25
@@ -20,28 +20,3 @@ BUILD_INTERRUPT(kvm_posted_intr_wakeup_ipi, POSTED_INTR_WAKEUP_VECTOR) BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR)#endif -/* -* every pentium local APIC has two 'local interrupts', with a -* soft-definable vector attached to both interrupts, one of -* which is a timer interrupt, the other one is error counter -* overflow. Linux uses the local APIC timer interrupt to get -* a much simpler SMP time architecture: -*/ -#ifdef CONFIG_X86_LOCAL_APIC - -#ifdef CONFIG_IRQ_WORK -BUILD_INTERRUPT(irq_work_interrupt, IRQ_WORK_VECTOR) -#endif - -#ifdef CONFIG_X86_THERMAL_VECTOR -BUILD_INTERRUPT(thermal_interrupt,THERMAL_APIC_VECTOR) -#endif - -#ifdef CONFIG_X86_MCE_THRESHOLD -BUILD_INTERRUPT(threshold_interrupt,THRESHOLD_APIC_VECTOR) -#endif - -#ifdef CONFIG_X86_MCE_AMD -BUILD_INTERRUPT(deferred_error_interrupt, DEFERRED_ERROR_VECTOR) -#endif -#endif
arch/x86/include/asm/hw_irq.h +0 -6
@@ -32,15 +32,9 @@ extern asmlinkage void kvm_posted_intr_ipi(void); extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); extern asmlinkage void kvm_posted_intr_nested_ipi(void); -extern asmlinkage void irq_work_interrupt(void); -extern asmlinkage void uv_bau_message_intr1(void); -extern asmlinkage void thermal_interrupt(void); extern asmlinkage void reschedule_interrupt(void); -extern asmlinkage void threshold_interrupt(void); -extern asmlinkage void deferred_error_interrupt(void); -#ifdef CONFIG_X86_LOCAL_APIC struct irq_data ;struct pci_dev ;
arch/x86/include/asm/idtentry.h +22 -0
@@ -582,6 +582,28 @@ DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR, sysvec_call_function_singleDECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_VECTOR , sysvec_call_function);#endif +#ifdef CONFIG_X86_LOCAL_APIC +# ifdef CONFIG_X86_UV +DECLARE_IDTENTRY_SYSVEC (UV_BAU_MESSAGE , sysvec_uv_bau_message); +# endif + +# ifdef CONFIG_X86_MCE_THRESHOLD +DECLARE_IDTENTRY_SYSVEC (THRESHOLD_APIC_VECTOR , sysvec_threshold); +# endif + +# ifdef CONFIG_X86_MCE_AMD +DECLARE_IDTENTRY_SYSVEC (DEFERRED_ERROR_VECTOR , sysvec_deferred_error); +# endif + +# ifdef CONFIG_X86_THERMAL_VECTOR +DECLARE_IDTENTRY_SYSVEC (THERMAL_APIC_VECTOR , sysvec_thermal); +# endif + +# ifdef CONFIG_IRQ_WORK +DECLARE_IDTENTRY_SYSVEC (IRQ_WORK_VECTOR , sysvec_irq_work); +# endif +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq_work.h +0 -1
@@ -10,7 +10,6 @@ static inline bool arch_irq_work_has_interrupt(void) return boot_cpu_has(X86_FEATURE_APIC); }extern void arch_irq_work_raise (void ); -extern __visible void smp_irq_work_interrupt(struct pt_regs *regs);#else static inline bool arch_irq_work_has_interrupt (void ) {
arch/x86/include/asm/traps.h +0 -5
@@ -34,11 +34,6 @@ static inline int get_si_code(unsigned long condition)extern int panic_on_unrecovered_nmi;void math_emulate (struct math_emu_info *); -#ifndef CONFIG_X86_32 -asmlinkage void smp_thermal_interrupt(struct pt_regs *regs); -asmlinkage void smp_threshold_interrupt(struct pt_regs *regs); -asmlinkage void smp_deferred_error_interrupt(struct pt_regs *regs); -#endif #ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message,
arch/x86/include/asm/uv/uv_bau.h +2 -6
@@ -12,6 +12,8 @@#define _ASM_X86_UV_UV_BAU_H#include <linux/bitmap.h> +#include <asm/idtentry.h> +#define BITSPERBYTE 8/*
@@ -799,12 +801,6 @@ static inline void bau_cpubits_clear(struct bau_local_cpumask *dstp, int nbits)bitmap_zero (&dstp->bits, nbits); } -extern void uv_bau_message_intr1 (void ); -#ifdef CONFIG_TRACING -#define trace_uv_bau_message_intr1 uv_bau_message_intr1 -#endif -extern void uv_bau_timeout_intr1 (void ); -struct atomic_short {short counter; };
arch/x86/kernel/cpu/mce/amd.c +2 -3
@@ -907,14 +907,13 @@ static void __log_error(unsigned int bank, u64 status, u64 addr, u64 misc)mce_log (&m); } -asmlinkage __visible void __irq_entry smp_deferred_error_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_deferred_error) { -entering_irq ();trace_deferred_error_apic_entry (DEFERRED_ERROR_VECTOR );inc_irq_stat (irq_deferred_error_count);deferred_error_int_vector ();trace_deferred_error_apic_exit (DEFERRED_ERROR_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }/*
arch/x86/kernel/cpu/mce/therm_throt.c +2 -3
@@ -614,14 +614,13 @@ static void unexpected_thermal_interrupt(void)static void (*smp_thermal_vector)(void ) = unexpected_thermal_interrupt; -asmlinkage __visible void __irq_entry smp_thermal_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_thermal) { -entering_irq ();trace_thermal_apic_entry (THERMAL_APIC_VECTOR );inc_irq_stat (irq_thermal_count);smp_thermal_vector ();trace_thermal_apic_exit (THERMAL_APIC_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }/* Thermal monitoring depends on APIC, ACPI and clock modulation */
arch/x86/kernel/cpu/mce/threshold.c +2 -3
@@ -21,12 +21,11 @@ static void default_threshold_interrupt(void)void (*mce_threshold_vector)(void ) = default_threshold_interrupt; -asmlinkage __visible void __irq_entry smp_threshold_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_threshold) { -entering_irq ();trace_threshold_apic_entry (THRESHOLD_APIC_VECTOR );inc_irq_stat (irq_threshold_count);mce_threshold_vector ();trace_threshold_apic_exit (THRESHOLD_APIC_VECTOR ); -exiting_ack_irq (); +ack_APIC_irq (); }
arch/x86/kernel/idt.c +14 -14
@@ -117,33 +117,33 @@ static const __initconst struct idt_data apic_idts[] = {#endif #ifdef CONFIG_X86_THERMAL_VECTOR - INTG(THERMAL_APIC_VECTOR, thermal_interrupt), + INTG(THERMAL_APIC_VECTOR, asm_sysvec_thermal),#endif #ifdef CONFIG_X86_MCE_THRESHOLD - INTG(THRESHOLD_APIC_VECTOR, threshold_interrupt), + INTG(THRESHOLD_APIC_VECTOR, asm_sysvec_threshold),#endif #ifdef CONFIG_X86_MCE_AMD - INTG(DEFERRED_ERROR_VECTOR, deferred_error_interrupt), + INTG(DEFERRED_ERROR_VECTOR, asm_sysvec_deferred_error),#endif #ifdef CONFIG_X86_LOCAL_APIC - INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), - INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi), + INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), + INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi), # ifdef CONFIG_HAVE_KVM - INTG(POSTED_INTR_VECTOR, kvm_posted_intr_ipi), - INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi), - INTG(POSTED_INTR_NESTED_VECTOR, kvm_posted_intr_nested_ipi), + INTG(POSTED_INTR_VECTOR, kvm_posted_intr_ipi), + INTG(POSTED_INTR_WAKEUP_VECTOR, kvm_posted_intr_wakeup_ipi), + INTG(POSTED_INTR_NESTED_VECTOR, kvm_posted_intr_nested_ipi), # endif # ifdef CONFIG_IRQ_WORK - INTG(IRQ_WORK_VECTOR, irq_work_interrupt), + INTG(IRQ_WORK_VECTOR, asm_sysvec_irq_work), # endif -#ifdef CONFIG_X86_UV - INTG(UV_BAU_MESSAGE, uv_bau_message_intr1), -#endif - INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), - INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), +# ifdef CONFIG_X86_UV + INTG(UV_BAU_MESSAGE, asm_sysvec_uv_bau_message), +# endif + INTG(SPURIOUS_APIC_VECTOR, asm_sysvec_spurious_apic_interrupt), + INTG(ERROR_APIC_VECTOR, asm_sysvec_error_interrupt), #endif };
arch/x86/kernel/irq_work.c +3 -3
@@ -9,18 +9,18 @@#include <linux/irq_work.h> #include <linux/hardirq.h> #include <asm/apic.h> +#include <asm/idtentry.h> #include <asm/trace/irq_vectors.h> #include <linux/interrupt.h> #ifdef CONFIG_X86_LOCAL_APIC -__visible void __irq_entry smp_irq_work_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_irq_work) { -ipi_entering_ack_irq (); +ack_APIC_irq ();trace_irq_work_entry (IRQ_WORK_VECTOR );inc_irq_stat (apic_irq_work_irqs);irq_work_run ();trace_irq_work_exit (IRQ_WORK_VECTOR ); -exiting_irq (); }void arch_irq_work_raise (void )
arch/x86/platform/uv/tlb_uv.c +1 -1
@@ -1272,7 +1272,7 @@ static void process_uv2_message(struct msg_desc *mdp, struct bau_control *bcp)* (the resource will not be freed until noninterruptable cpus see this * interrupt; hardware may timeout the s/w ack and reply ERROR) */ -void uv_bau_message_interrupt (struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_uv_bau_message) {int count =0 ;cycles_t time_start;
Re: [patch V6 26/37] x86/entry: Convert various system vectors
Quoted 10 lines · Thomas Gleixner
Convert various system vectors to
IDTENTRY_SYSVEC
- Implement the C entry point withDEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub withDECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove theBUILD_INTERRUPTentries in 32bit
- Remove the old prototypesNo functional change.
PATCH v6 27/37 x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC
In reply to Thomas GleixnerConvert KVM specific system vectors to IDTENTRY_SYSVEC*:
The two empty stub handlers which only increment the stats counter do no need to run on the interrupt stack. Use IDTENTRY_SYSVEC_DIRECT for them.
The wakeup handler does more work and runs on the interrupt stack.
None of these handlers need to save and restore the irq_regs pointer.
arch/x86/entry/entry_64.S +0 -7
@@ -956,13 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm - -#ifdef CONFIG_HAVE_KVM -apicinterrupt3 POSTED_INTR_VECTOR kvm_posted_intr_ipi smp_kvm_posted_intr_ipi -apicinterrupt3 POSTED_INTR_WAKEUP_VECTOR kvm_posted_intr_wakeup_ipi smp_kvm_posted_intr_wakeup_ipi -apicinterrupt3 POSTED_INTR_NESTED_VECTOR kvm_posted_intr_nested_ipi smp_kvm_posted_intr_nested_ipi -#endif - #ifdef CONFIG_SMP apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt #endif
arch/x86/include/asm/entry_arch.h +0 -7
@@ -13,10 +13,3 @@#ifdef CONFIG_SMP BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR)#endif - -#ifdef CONFIG_HAVE_KVM -BUILD_INTERRUPT(kvm_posted_intr_ipi, POSTED_INTR_VECTOR) -BUILD_INTERRUPT(kvm_posted_intr_wakeup_ipi, POSTED_INTR_WAKEUP_VECTOR) -BUILD_INTERRUPT(kvm_posted_intr_nested_ipi, POSTED_INTR_NESTED_VECTOR) -#endif -
arch/x86/include/asm/hw_irq.h +0 -4
@@ -29,10 +29,6 @@#include <asm/sections.h> /* Interrupt handlers registered during init_IRQ */ -extern asmlinkage void kvm_posted_intr_ipi(void); -extern asmlinkage void kvm_posted_intr_wakeup_ipi(void); -extern asmlinkage void kvm_posted_intr_nested_ipi(void); - extern asmlinkage void reschedule_interrupt(void);#ifdef CONFIG_X86_LOCAL_APIC
arch/x86/include/asm/idtentry.h +6 -0
@@ -604,6 +604,12 @@ DECLARE_IDTENTRY_SYSVEC(IRQ_WORK_VECTOR, sysvec_irq_work);# endif #endif +#ifdef CONFIG_HAVE_KVM +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_VECTOR , sysvec_kvm_posted_intr_ipi); +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_WAKEUP_VECTOR , sysvec_kvm_posted_intr_wakeup_ipi); +DECLARE_IDTENTRY_SYSVEC (POSTED_INTR_NESTED_VECTOR , sysvec_kvm_posted_intr_nested_ipi); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/irq.h +0 -3
@@ -26,9 +26,6 @@ extern void fixup_irqs(void);#ifdef CONFIG_HAVE_KVM extern void kvm_set_posted_intr_wakeup_handler (void (*handler)(void )); -extern __visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs); -extern __visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs); -extern __visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs);#endif extern void (*x86_platform_ipi_callback)(void );
arch/x86/kernel/idt.c +3 -3
@@ -132,9 +132,9 @@ static const __initconst struct idt_data apic_idts[] = { INTG(LOCAL_TIMER_VECTOR, asm_sysvec_apic_timer_interrupt), INTG(X86_PLATFORM_IPI_VECTOR, asm_sysvec_x86_platform_ipi),# ifdef CONFIG_HAVE_KVM -INTG (POSTED_INTR_VECTOR , kvm_posted_intr_ipi), -INTG (POSTED_INTR_WAKEUP_VECTOR , kvm_posted_intr_wakeup_ipi), -INTG (POSTED_INTR_NESTED_VECTOR , kvm_posted_intr_nested_ipi), +INTG (POSTED_INTR_VECTOR , asm_sysvec_kvm_posted_intr_ipi), +INTG (POSTED_INTR_WAKEUP_VECTOR , asm_sysvec_kvm_posted_intr_wakeup_ipi), +INTG (POSTED_INTR_NESTED_VECTOR , asm_sysvec_kvm_posted_intr_nested_ipi), # endif # ifdef CONFIG_IRQ_WORK INTG(IRQ_WORK_VECTOR, asm_sysvec_irq_work),
arch/x86/kernel/irq.c +6 -18
@@ -290,41 +290,29 @@ EXPORT_SYMBOL_GPL(kvm_set_posted_intr_wakeup_handler);/* * Handler for POSTED_INTERRUPT_VECTOR. */ -__visible void smp_kvm_posted_intr_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC_SIMPLE (sysvec_kvm_posted_intr_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_ipis); -exiting_irq (); -set_irq_regs (old_regs); }/* * Handler for POSTED_INTERRUPT_WAKEUP_VECTOR. */ -__visible void smp_kvm_posted_intr_wakeup_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_kvm_posted_intr_wakeup_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_wakeup_ipis);kvm_posted_intr_wakeup_handler (); -exiting_irq (); -set_irq_regs (old_regs); }/* * Handler for POSTED_INTERRUPT_NESTED_VECTOR. */ -__visible void smp_kvm_posted_intr_nested_ipi(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC_SIMPLE (sysvec_kvm_posted_intr_nested_ipi) { -struct pt_regs *old_regs =set_irq_regs (regs); - -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (kvm_posted_intr_nested_ipis); -exiting_irq (); -set_irq_regs (old_regs); }#endif
Re: [patch V6 27/37] x86/entry: Convert KVM vectors to IDTENTRY_SYSVEC
Quoted 10 lines · Thomas Gleixner
Convert KVM specific system vectors to IDTENTRY_SYSVEC*:
The two empty stub handlers which only increment the stats counter do no need to run on the interrupt stack. Use
IDTENTRY_SYSVEC_DIRECTfor them.The wakeup handler does more work and runs on the interrupt stack.
None of these handlers need to save and restore the
irq_regspointer.
PATCH v6 28/37 x86/entry: Convert various hypervisor vectors to IDTENTRY_SYSVEC
In reply to Thomas GleixnerConvert various hypervisor vectors to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
No functional change.
arch/x86/entry/entry_32.S +0 -14
@@ -1342,20 +1342,6 @@ BUILD_INTERRUPT3(xen_hvm_callback_vector, HYPERVISOR_CALLBACK_VECTOR, xen_evtchn_do_upcall) #endif - -#if IS_ENABLED(CONFIG_HYPERV) - -BUILD_INTERRUPT3(hyperv_callback_vector, HYPERVISOR_CALLBACK_VECTOR, - hyperv_vector_handler) - -BUILD_INTERRUPT3(hyperv_reenlightenment_vector, HYPERV_REENLIGHTENMENT_VECTOR, - hyperv_reenlightenment_intr) - -BUILD_INTERRUPT3(hv_stimer0_callback_vector, HYPERV_STIMER0_VECTOR, - hv_stimer0_vector_handler) - -#endif /* CONFIG_HYPERV */ - SYM_CODE_START_LOCAL_NOALIGN(handle_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -17
@@ -1116,23 +1116,6 @@ apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \ xen_hvm_callback_vector xen_evtchn_do_upcall #endif - -#if IS_ENABLED(CONFIG_HYPERV) -apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \ - hyperv_callback_vector hyperv_vector_handler - -apicinterrupt3 HYPERV_REENLIGHTENMENT_VECTOR \ - hyperv_reenlightenment_vector hyperv_reenlightenment_intr - -apicinterrupt3 HYPERV_STIMER0_VECTOR \ - hv_stimer0_callback_vector hv_stimer0_vector_handler -#endif /* CONFIG_HYPERV */ - -#if IS_ENABLED(CONFIG_ACRN_GUEST) -apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \ - acrn_hv_callback_vector acrn_hv_vector_handler -#endif - /* * Save all registers in pt_regs, and switch gs if needed. * Use slow, but surefire "are we in kernel?" check.
arch/x86/hyperv/hv_init.c +3 -6
@@ -15,6 +15,7 @@#include <asm/hypervisor.h> #include <asm/hyperv-tlfs.h> #include <asm/mshyperv.h> +#include <asm/idtentry.h> #include <linux/version.h> #include <linux/vmalloc.h> #include <linux/mm.h>
@@ -153,15 +154,11 @@ static inline bool hv_reenlightenment_available(void) ms_hyperv.features &HV_X64_ACCESS_REENLIGHTENMENT ; } -__visible void __irq_entry hyperv_reenlightenment_intr(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_hyperv_reenlightenment) { -entering_ack_irq (); - +ack_APIC_irq ();inc_irq_stat (irq_hv_reenlightenment_count); -schedule_delayed_work (&hv_reenlightenment_work,HZ /10 ); - -exiting_irq (); }void set_hv_tscchange_cb (void (*cb)(void ))
arch/x86/include/asm/acrn.h deleted +0 -11
@@ -1,11 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_X86_ACRN_H -#define _ASM_X86_ACRN_H - -extern void acrn_hv_callback_vector (void ); -#ifdef CONFIG_TRACING -#define trace_acrn_hv_callback_vector acrn_hv_callback_vector -#endif - -extern void acrn_hv_vector_handler (struct pt_regs *regs); -#endif /* _ASM_X86_ACRN_H */
arch/x86/include/asm/apic.h +0 -20
@@ -519,26 +519,6 @@ static inline bool apic_id_is_primary_thread(unsigned int id) { return false; }static inline void apic_smt_update (void ) { }#endif -extern void irq_enter (void ); -extern void irq_exit (void ); - -static inline void entering_irq (void ) -{ -irq_enter (); -kvm_set_cpu_l1tf_flush_l1d (); -} - -static inline void entering_ack_irq (void ) -{ -entering_irq (); -ack_APIC_irq (); -} - -static inline void exiting_irq (void ) -{ -irq_exit (); -} -extern void ioapic_zap_locks (void );#endif /* _ASM_X86_APIC_H */
arch/x86/include/asm/idtentry.h +10 -0
@@ -610,6 +610,16 @@ DECLARE_IDTENTRY_SYSVEC(POSTED_INTR_WAKEUP_VECTOR, sysvec_kvm_posted_intr_wakeupDECLARE_IDTENTRY_SYSVEC (POSTED_INTR_NESTED_VECTOR , sysvec_kvm_posted_intr_nested_ipi);#endif +#if IS_ENABLED (CONFIG_HYPERV ) +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_CALLBACK_VECTOR , sysvec_hyperv_callback); +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_REENLIGHTENMENT_VECTOR , sysvec_hyperv_reenlightenment); +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_STIMER0_VECTOR , sysvec_hyperv_stimer0); +#endif + +#if IS_ENABLED (CONFIG_ACRN_GUEST ) +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_CALLBACK_VECTOR , sysvec_acrn_hv_callback); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/include/asm/mshyperv.h +0 -13
@@ -54,20 +54,8 @@ typedef int (*hyperv_fill_flush_list_func)(vclocks_set_used (VDSO_CLOCKMODE_HVCLOCK );#define hv_get_raw_timer () rdtsc_ordered() -void hyperv_callback_vector (void ); -void hyperv_reenlightenment_vector (void ); -#ifdef CONFIG_TRACING -#define trace_hyperv_callback_vector hyperv_callback_vector -#endif void hyperv_vector_handler (struct pt_regs *regs); -/* -* Routines for stimer0 Direct Mode handling. -* On x86/x64, there are no percpu actions to take. -*/ -void hv_stimer0_vector_handler (struct pt_regs *regs); -void hv_stimer0_callback_vector (void ); -static inline void hv_enable_stimer0_percpu_irq (int irq) {}static inline void hv_disable_stimer0_percpu_irq (int irq) {}
@@ -226,7 +214,6 @@ void hyperv_setup_mmu_ops(void);void *hv_alloc_hyperv_page (void );void *hv_alloc_hyperv_zeroed_page (void );void hv_free_hyperv_page (unsigned long addr); -void hyperv_reenlightenment_intr (struct pt_regs *regs);void set_hv_tscchange_cb (void (*cb)(void ));void clear_hv_tscchange_cb (void );void hyperv_stop_tsc_emulation (void );
arch/x86/kernel/cpu/acrn.c +4 -5
@@ -10,10 +10,10 @@*/ #include <linux/interrupt.h> -#include <asm/acrn.h> #include <asm/apic.h> #include <asm/desc.h> #include <asm/hypervisor.h> +#include <asm/idtentry.h> #include <asm/irq_regs.h> static uint32_t __init acrn_detect (void )
@@ -24,7 +24,7 @@ static uint32_t __init acrn_detect(void)static void __init acrn_init_platform (void ) {/* Setup the IDT for ACRN hypervisor callback */ -alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , acrn_hv_callback_vector); +alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , asm_sysvec_acrn_hv_callback); }static bool acrn_x2apic_available (void )
@@ -39,7 +39,7 @@ static bool acrn_x2apic_available(void)static void (*acrn_intr_handler)(void ); -__visible void __irq_entry acrn_hv_vector_handler(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_acrn_hv_callback) {struct pt_regs *old_regs =set_irq_regs (regs);
@@ -50,13 +50,12 @@ __visible void __irq_entry acrn_hv_vector_handler(struct pt_regs *regs)* will block the interrupt whose vector is lower than * HYPERVISOR_CALLBACK_VECTOR. */ -entering_ack_irq (); +ack_APIC_irq ();inc_irq_stat (irq_hv_callback_count);if (acrn_intr_handler)acrn_intr_handler (); -exiting_irq ();set_irq_regs (old_regs); }
arch/x86/kernel/cpu/mshyperv.c +10 -12
@@ -23,6 +23,7 @@#include <asm/hyperv-tlfs.h> #include <asm/mshyperv.h> #include <asm/desc.h> +#include <asm/idtentry.h> #include <asm/irq_regs.h> #include <asm/i8259.h> #include <asm/apic.h>
@@ -40,11 +41,10 @@ static void (*hv_stimer0_handler)(void);static void (*hv_kexec_handler)(void );static void (*hv_crash_handler)(struct pt_regs *regs); -__visible void __irq_entry hyperv_vector_handler(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_hyperv_callback) {struct pt_regs *old_regs =set_irq_regs (regs); -entering_irq ();inc_irq_stat (irq_hv_callback_count);if (vmbus_handler)vmbus_handler ();
@@ -52,7 +52,6 @@ __visible void __irq_entry hyperv_vector_handler(struct pt_regs *regs)if (ms_hyperv.hints &HV_DEPRECATING_AEOI_RECOMMENDED )ack_APIC_irq (); -exiting_irq ();set_irq_regs (old_regs); }
@@ -73,19 +72,16 @@ EXPORT_SYMBOL_GPL(hv_remove_vmbus_irq);* Routines to do per-architecture handling of stimer0 * interrupts when in Direct Mode */ - -__visible void __irq_entry hv_stimer0_vector_handler(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC (sysvec_hyperv_stimer0) {struct pt_regs *old_regs =set_irq_regs (regs); -entering_irq ();inc_irq_stat (hyperv_stimer0_count);if (hv_stimer0_handler)hv_stimer0_handler ();add_interrupt_randomness (HYPERV_STIMER0_VECTOR ,0 );ack_APIC_irq (); -exiting_irq ();set_irq_regs (old_regs); }
@@ -331,17 +327,19 @@ static void __init ms_hyperv_init_platform(void) x86_platform.apic_post_init = hyperv_init;hyperv_setup_mmu_ops ();/* Setup the IDT for hypervisor callback */ -alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , hyperv_callback_vector); +alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , sysvec_hyperv_callback);/* Setup the IDT for reenlightenment notifications */ -if (ms_hyperv.features &HV_X64_ACCESS_REENLIGHTENMENT ) +if (ms_hyperv.features &HV_X64_ACCESS_REENLIGHTENMENT ) {alloc_intr_gate (HYPERV_REENLIGHTENMENT_VECTOR , - hyperv_reenlightenment_vector); + asm_sysvec_hyperv_reenlightenment); + }/* Setup the IDT for stimer0 */ -if (ms_hyperv.misc_features &HV_STIMER_DIRECT_MODE_AVAILABLE ) +if (ms_hyperv.misc_features &HV_STIMER_DIRECT_MODE_AVAILABLE ) {alloc_intr_gate (HYPERV_STIMER0_VECTOR , - hv_stimer0_callback_vector); + asm_sysvec_hyperv_stimer0); + }# ifdef CONFIG_SMP smp_ops.smp_prepare_boot_cpu = hv_smp_prepare_boot_cpu;
Re: [patch V6 28/37] x86/entry: Convert various hypervisor vectors to IDTENTRY_SYSVEC
Quoted 10 lines · Thomas Gleixner
Convert various hypervisor vectors to
IDTENTRY_SYSVEC
- Implement the C entry point withDEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub withDECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove theBUILD_INTERRUPTentries in 32bit
- Remove the old prototypesNo functional change.
PATCH v6 29/37 x86/entry: Convert XEN hypercall vector to IDTENTRY_SYSVEC
In reply to Thomas GleixnerConvert the last oldstyle defined vector to IDTENTRY_SYSVEC
- Implement the C entry point with DEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub with DECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove the BUILD_INTERRUPT entries in 32bit
- Remove the old prototypes
Fixup the related XEN code by providing the primary C entry point in x86 to avoid cluttering the generic code with X86'isms.
No functional change.
--- arch/x86/entry/entry_32.S | 5 ----- arch/x86/entry/entry_64.S | 5 ----- arch/x86/include/asm/idtentry.h | 4 ++++ arch/x86/xen/enlighten_hvm.c | 12 ++++++++++++ drivers/xen/events/events_base.c | 6 ++---- include/xen/events.h | 7 ------- 6 files changed, 18 insertions(+), 21 deletions(-)
arch/x86/entry/entry_32.S +0 -5
@@ -1337,11 +1337,6 @@ SYM_FUNC_START(xen_failsafe_callback) SYM_FUNC_END(xen_failsafe_callback) #endif /* CONFIG_XEN_PV */ -#ifdef CONFIG_XEN_PVHVM -BUILD_INTERRUPT3(xen_hvm_callback_vector, HYPERVISOR_CALLBACK_VECTOR, - xen_evtchn_do_upcall) -#endif - SYM_CODE_START_LOCAL_NOALIGN(handle_exception) /* the function address is in %gs's slot on the stack */ SAVE_ALL switch_stacks=1 skip_gs=1 unwind_espfix=1
arch/x86/entry/entry_64.S +0 -5
@@ -1111,11 +1111,6 @@ SYM_CODE_START(xen_failsafe_callback) SYM_CODE_END(xen_failsafe_callback) #endif /* CONFIG_XEN_PV */ -#ifdef CONFIG_XEN_PVHVM -apicinterrupt3 HYPERVISOR_CALLBACK_VECTOR \ - xen_hvm_callback_vector xen_evtchn_do_upcall -#endif - /* * Save all registers in pt_regs, and switch gs if needed. * Use slow, but surefire "are we in kernel?" check.
arch/x86/include/asm/idtentry.h +4 -0
@@ -620,6 +620,10 @@ DECLARE_IDTENTRY_SYSVEC(HYPERVISOR_STIMEDECLARE_IDTENTRY_SYSVEC (HYPERVISOR_CALLBACK_VECTOR , sysvec_acrn_hv_callback);#endif +#ifdef CONFIG_XEN_PVHVM +DECLARE_IDTENTRY_SYSVEC (HYPERVISOR_CALLBACK_VECTOR , sysvec_xen_hvm_callback); +#endif +#undef X86_TRAP_OTHER#endif
arch/x86/xen/enlighten_hvm.c +12 -0
@@ -13,6 +13,7 @@#include <asm/smp.h> #include <asm/reboot.h> #include <asm/setup.h> +#include <asm/idtentry.h> #include <asm/hypervisor.h> #include <asm/e820/api.h> #include <asm/early_ioremap.h>
@@ -118,6 +119,17 @@ static void __init init_hvm_pv_info(voidthis_cpu_write (xen_vcpu_id,smp_processor_id ()); } +DEFINE_IDTENTRY_SYSVEC (sysvec_xen_hvm_callback) +{ +struct pt_regs *old_regs =set_irq_regs (regs); + +inc_irq_stat (irq_hv_callback_count); + +xen_hvm_evtchn_do_upcall (); + +set_irq_regs (old_regs); +} +#ifdef CONFIG_KEXEC_CORE static void xen_hvm_shutdown (void ) {
drivers/xen/events/events_base.c +2 -4
@@ -37,6 +37,7 @@#ifdef CONFIG_X86 #include <asm/desc.h> #include <asm/ptrace.h> +#include <asm/idtentry.h> #include <asm/irq.h> #include <asm/io_apic.h> #include <asm/i8259.h>
@@ -1236,9 +1237,6 @@ void xen_evtchn_do_upcall(struct pt_regsstruct pt_regs *old_regs =set_irq_regs (regs);irq_enter (); -#ifdef CONFIG_X86 -inc_irq_stat (irq_hv_callback_count); -#endif __xen_evtchn_do_upcall ();
@@ -1658,7 +1656,7 @@ static __init void xen_alloc_callback_vereturn ;pr_info ("Xen HVM callback vector for event delivery is enabled\n" ); - alloc_intr_gate(HYPERVISOR_CALLBACK_VECTOR, xen_hvm_callback_vector); + alloc_intr_gate(HYPERVISOR_CALLBACK_VECTOR, asm_sysvec_xen_hvm_callback); }#else void xen_setup_callback_vector (void ) {}
include/xen/events.h +0 -7
@@ -90,13 +90,6 @@ unsigned int irq_from_evtchn(evtchn_portint irq_from_virq (unsigned int cpu,unsigned int virq);evtchn_port_t evtchn_from_irq (unsigned irq); -#ifdef CONFIG_XEN_PVHVM -/* Xen HVM evtchn vector callback */ -void xen_hvm_callback_vector (void ); -#ifdef CONFIG_TRACING -#define trace_xen_hvm_callback_vector xen_hvm_callback_vector -#endif -#endif int xen_set_callback_via (uint64_t via);void xen_evtchn_do_upcall (struct pt_regs *regs);void xen_hvm_evtchn_do_upcall (void );
Re: [patch V6 29/37] x86/entry: Convert XEN hypercall vector to IDTENTRY_SYSVEC
Quoted 8 lines · Thomas Gleixner
Convert the last oldstyle defined vector to
IDTENTRY_SYSVEC
- Implement the C entry point withDEFINE_IDTENTRY_SYSVEC
- Emit the ASM stub withDECLARE_IDTENTRY_SYSVEC
- Remove the ASM idtentries in 64bit
- Remove theBUILD_INTERRUPTentries in 32bit
- Remove the old prototypes
PATCH v6 30/37 x86/entry: Convert reschedule interrupt to IDTENTRY_RAW
In reply to Thomas GleixnerThe scheduler IPI does not need the full interrupt entry handling logic when the entry is from kernel mode.
Even if tracing is enabled the only requirement is that RCU is watching and preempt_count has the hardirq bit on.
The NOHZ tick state does not have to be adjusted. If the tick is not running then the CPU is in idle and the idle exit will restore the tick. Softinterrupts are not raised here, so handling them on return is not required either.
User mode entry must go through the regular entry path as it will invoke the scheduler on return so context tracking needs to be in the correct state.
Use IDTENTRY_RAW and the RCU conditional variants of idtentry_enter/exit() to guarantee that RCU is watching even if the IPI hits a RCU idle section.
Remove the tracepoint static key conditional which is incomplete vs. tracing anyway because e.g. ack_APIC_irq() calls out into instrumentable code.
Avoid the overhead of irq time accounting and introduce variants of __irq_enter/exit() so instrumentation observes the correct preempt count state.
Spare the switch to the interrupt stack as the IPI is not going to use only a minimal amount of stack space.
arch/x86/entry/entry_64.S +0 -4
@@ -956,10 +956,6 @@ apicinterrupt3 \num \sym \do_sym POP_SECTION_IRQENTRY .endm -#ifdef CONFIG_SMP -apicinterrupt RESCHEDULE_VECTOR reschedule_interrupt smp_reschedule_interrupt -#endif - /* * Reload gs selector with exception handling * edi: new selector
arch/x86/include/asm/entry_arch.h +0 -3
@@ -10,6 +10,3 @@* is no hardware IRQ pin equivalent for them, they are triggered * through the ICC by us (IPIs) */ -#ifdef CONFIG_SMP -BUILD_INTERRUPT(reschedule_interrupt,RESCHEDULE_VECTOR) -#endif
arch/x86/include/asm/hw_irq.h +0 -3
@@ -28,9 +28,6 @@#include <asm/irq.h> #include <asm/sections.h> -/* Interrupt handlers registered during init_IRQ */ -extern asmlinkage void reschedule_interrupt(void); -#ifdef CONFIG_X86_LOCAL_APIC struct irq_data ;struct pci_dev ;
arch/x86/include/asm/idtentry.h +1 -0
@@ -576,6 +576,7 @@ DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR, sysvec_x86_platform_ipi);#endif #ifdef CONFIG_SMP +DECLARE_IDTENTRY (RESCHEDULE_VECTOR , sysvec_reschedule_ipi);DECLARE_IDTENTRY_SYSVEC (IRQ_MOVE_CLEANUP_VECTOR , sysvec_irq_move_cleanup);DECLARE_IDTENTRY_SYSVEC (REBOOT_VECTOR , sysvec_reboot);DECLARE_IDTENTRY_SYSVEC (CALL_FUNCTION_SINGLE_VECTOR , sysvec_call_function_single);
arch/x86/include/asm/trace/common.h +0 -4
@@ -5,12 +5,8 @@DECLARE_STATIC_KEY_FALSE (trace_pagefault_key);#define trace_pagefault_enabled () \ static_branch_unlikely(&trace_pagefault_key) -DECLARE_STATIC_KEY_FALSE (trace_resched_ipi_key); -#define trace_resched_ipi_enabled () \ - static_branch_unlikely(&trace_resched_ipi_key)#else static inline bool trace_pagefault_enabled (void ) {return false; } -static inline bool trace_resched_ipi_enabled (void ) {return false; }#endif #endif
arch/x86/include/asm/trace/irq_vectors.h +1 -16
@@ -10,9 +10,6 @@#ifdef CONFIG_X86_LOCAL_APIC -extern int trace_resched_ipi_reg (void ); -extern void trace_resched_ipi_unreg (void ); - DECLARE_EVENT_CLASS(x86_irq_vector, TP_PROTO(int vector),
@@ -37,18 +34,6 @@ DEFINE_EVENT_FN(x86_irq_vector, name##_exit, \ TP_PROTO(int vector), \ TP_ARGS(vector), NULL, NULL); -#define DEFINE_RESCHED_IPI_EVENT (name) \ -DEFINE_EVENT_FN(x86_irq_vector, name##_entry, \ - TP_PROTO(int vector), \ - TP_ARGS(vector), \ - trace_resched_ipi_reg, \ - trace_resched_ipi_unreg); \ -DEFINE_EVENT_FN(x86_irq_vector, name##_exit, \ - TP_PROTO(int vector), \ - TP_ARGS(vector), \ - trace_resched_ipi_reg, \ - trace_resched_ipi_unreg); -/* * local_timer - called when entering/exiting a local timer interrupt * vector handler
@@ -99,7 +84,7 @@ TRACE_EVENT_PERF_PERM(irq_work_exit, is_sampling_event(p_event) ? -EPERM : 0);/* * reschedule - called when entering/exiting a reschedule vector handler */ -DEFINE_RESCHED_IPI_EVENT (reschedule); +DEFINE_IRQ_VECTOR_EVENT (reschedule);/* * call_function - called when entering/exiting a call function interrupt
arch/x86/kernel/idt.c +1 -1
@@ -109,7 +109,7 @@ static const __initconst struct idt_data def_idts[] = {*/ static const __initconst struct idt_data apic_idts[] = { #ifdef CONFIG_SMP -INTG (RESCHEDULE_VECTOR , reschedule_interrupt), +INTG (RESCHEDULE_VECTOR , asm_sysvec_reschedule_ipi),INTG (CALL_FUNCTION_VECTOR , asm_sysvec_call_function),INTG (CALL_FUNCTION_SINGLE_VECTOR , asm_sysvec_call_function_single), INTG(IRQ_MOVE_CLEANUP_VECTOR, asm_sysvec_irq_move_cleanup),
arch/x86/kernel/smp.c +4 -15
@@ -220,26 +220,15 @@ static void native_stop_other_cpus(int wait)/* * Reschedule call back. KVM uses this interrupt to force a cpu out of -* guest mode +* guest mode. */ -__visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs) +DEFINE_IDTENTRY_SYSVEC_SIMPLE (sysvec_reschedule_ipi) {ack_APIC_irq (); +trace_reschedule_entry (RESCHEDULE_VECTOR );inc_irq_stat (irq_resched_count); - -if (trace_resched_ipi_enabled ()) { -/* -* scheduler_ipi() might call irq_enter() as well, but -* nested calls are fine. -*/ -irq_enter (); -trace_reschedule_entry (RESCHEDULE_VECTOR ); -scheduler_ipi (); -trace_reschedule_exit (RESCHEDULE_VECTOR ); -irq_exit (); -return ; - }scheduler_ipi (); +trace_reschedule_exit (RESCHEDULE_VECTOR ); }DEFINE_IDTENTRY_SYSVEC (sysvec_call_function)
arch/x86/kernel/tracepoint.c +0 -17
@@ -25,20 +25,3 @@ void trace_pagefault_unreg(void) {static_branch_dec (&trace_pagefault_key); } - -#ifdef CONFIG_SMP - -DEFINE_STATIC_KEY_FALSE (trace_resched_ipi_key); - -int trace_resched_ipi_reg (void ) -{ -static_branch_inc (&trace_resched_ipi_key); -return 0 ; -} - -void trace_resched_ipi_unreg (void ) -{ -static_branch_dec (&trace_resched_ipi_key); -} - -#endif
Andy Lutomirski
maintainerRe: [patch V6 30/37] x86/entry: Convert reschedule interrupt to IDTENTRY_RAW
Quoted 20 lines · Thomas Gleixner
The scheduler IPI does not need the full interrupt entry handling logic when the entry is from kernel mode.
Even if tracing is enabled the only requirement is that RCU is watching and
preempt_counthas the hardirq bit on.The NOHZ tick state does not have to be adjusted. If the tick is not running then the CPU is in idle and the idle exit will restore the tick. Softinterrupts are not raised here, so handling them on return is not required either.
User mode entry must go through the regular entry path as it will invoke the scheduler on return so context tracking needs to be in the correct state.
Use
IDTENTRY_RAWand the RCU conditional variants of idtentry_enter/exit() to guarantee that RCU is watching even if the IPI hits a RCU idle section.
Quoted 7 lines
Remove the tracepoint static key conditional which is incomplete vs. tracing anyway because e.g.
ack_APIC_irq()calls out into instrumentable code.Avoid the overhead of irq time accounting and introduce variants of __irq_enter/exit() so instrumentation observes the correct preempt count state.
Leftover text from an old version?
The code is Reviewed-by: Andy Lutomirski <[email protected]>
Re: [patch V6 30/37] x86/entry: Convert reschedule interrupt to IDTENTRY_RAW
Quoted 25 lines · Andy Lutomirski
Quoted 24 lines · Thomas Gleixner
The scheduler IPI does not need the full interrupt entry handling logic when the entry is from kernel mode.
Even if tracing is enabled the only requirement is that RCU is watching and
preempt_counthas the hardirq bit on.The NOHZ tick state does not have to be adjusted. If the tick is not running then the CPU is in idle and the idle exit will restore the tick. Softinterrupts are not raised here, so handling them on return is not required either.
User mode entry must go through the regular entry path as it will invoke the scheduler on return so context tracking needs to be in the correct state.
Use
IDTENTRY_RAWand the RCU conditional variants of idtentry_enter/exit() to guarantee that RCU is watching even if the IPI hits a RCU idle section.Remove the tracepoint static key conditional which is incomplete vs. tracing anyway because e.g.
ack_APIC_irq()calls out into instrumentable code.
Quoted 5 lines
Quoted 3 lines
Avoid the overhead of irq time accounting and introduce variants of __irq_enter/exit() so instrumentation observes the correct preempt count state.
Leftover text from an old version?
Indeed
Quoted 1 line
The code is Reviewed-by: Andy Lutomirski <[email protected]>
PATCH v6 31/37 x86/entry: Remove the apic/BUILD interrupt leftovers
In reply to Thomas GleixnerRemove all the code which was there to emit the system vector stubs. All users are gone.
Move the now unused GET_CR2_INTO macro muck to head_64.S where the last user is. Fixup the eye hurting comment there while at it.
arch/x86/entry/calling.h +0 -20
@@ -351,23 +351,3 @@ For 32-bit we have the following conventions - kernel is built withcall stackleak_erase #endif .endm - -/* - * This does 'call enter_from_user_mode' unless we can avoid it based on - * kernel config or using the static jump infrastructure. - */ -.macro CALL_enter_from_user_mode -#ifdef CONFIG_CONTEXT_TRACKING -#ifdef CONFIG_JUMP_LABEL - STATIC_JUMP_IF_FALSE .Lafter_call_\@, context_tracking_key, def=0 -#endif - call enter_from_user_mode -.Lafter_call_\@: -#endif -.endm - -#ifdef CONFIG_PARAVIRT_XXL -#define GET_CR2_INTO(reg) GET_CR2_INTO_AX ; _ASM_MOV %_ASM_AX, reg -#else -#define GET_CR2_INTO(reg) _ASM_MOV %cr2, reg -#endif
arch/x86/entry/entry_32.S +0 -18
@@ -1233,24 +1233,6 @@ SYM_FUNC_END(entry_INT80_32) #endif .endm -#define BUILD_INTERRUPT3(name, nr, fn) \ -SYM_FUNC_START(name) \ - ASM_CLAC; \ - pushl $~(nr); \ - SAVE_ALL switch_stacks=1; \ - ENCODE_FRAME_POINTER; \ - TRACE_IRQS_OFF \ - movl %esp, %eax; \ - call fn; \ - jmp ret_from_intr; \ -SYM_FUNC_END(name) - -#define BUILD_INTERRUPT(name, nr) \ - BUILD_INTERRUPT3(name, nr, smp_##name); \ - -/* The include is where all of the SMP etc. interrupts come from */ -#include <asm/entry_arch.h> - #ifdef CONFIG_PARAVIRT SYM_CODE_START(native_iret) iret
arch/x86/entry/entry_64.S +0 -143
162 lines not on this page · raw message
arch/x86/include/asm/entry_arch.h deleted +0 -12
@@ -1,12 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* -* This file is designed to contain the BUILD_INTERRUPT specifications for -* all of the extra named interrupt vectors used by the architecture. -* Usually this is the Inter Process Interrupts (IPIs) -*/ - -/* -* The following vectors are part of the Linux architecture, there -* is no hardware IRQ pin equivalent for them, they are triggered -* through the ICC by us (IPIs) -*/
arch/x86/kernel/head_64.S +4 -3
@@ -29,15 +29,16 @@ #ifdef CONFIG_PARAVIRT_XXL #include <asm/asm-offsets.h> #include <asm/paravirt.h> +#define GET_CR2_INTO(reg) GET_CR2_INTO_AX ; _ASM_MOV %_ASM_AX, reg #else #define INTERRUPT_RETURN iretq +#define GET_CR2_INTO(reg) _ASM_MOV %cr2, reg #endif -/* we are not able to switch in one step to the final KERNEL ADDRESS SPACE +/* + * We are not able to switch in one step to the final KERNEL ADDRESS SPACE * because we need identity-mapped pages. - * */ - #define l4_index(x) (((x) >> 39) & 511) #define pud_index(x) (((x) >> PUD_SHIFT) & (PTRS_PER_PUD-1))
Re: [patch V6 31/37] x86/entry: Remove the apic/BUILD interrupt leftovers
Quoted 7 lines · Thomas Gleixner
Remove all the code which was there to emit the system vector stubs. All users are gone.
Move the now unused
GET_CR2_INTOmacro muck tohead_64.Swhere the last user is. Fixup the eye hurting comment there while at it.
PATCH v6 32/37 x86/entry/64: Remove IRQ stack switching ASM
In reply to Thomas Gleixnerarch/x86/entry/entry_64.S +0 -96
@@ -370,102 +370,6 @@ SYM_CODE_END(ret_from_fork) #endif .endm -/* - * Enters the IRQ stack if we're not already using it. NMI-safe. Clobbers - * flags and puts old RSP into old_rsp, and leaves all other GPRs alone. - * Requires kernel GSBASE. - * - * The invariant is that, if irq_count != -1, then the IRQ stack is in use. - */ -.macro ENTER_IRQ_STACK regs=1 old_rsp save_ret=0 - DEBUG_ENTRY_ASSERT_IRQS_OFF - - .if \save_ret - /* - * If save_ret is set, the original stack contains one additional - * entry -- the return address. Therefore, move the address one - * entry below %rsp to \old_rsp. - */ - leaq 8(%rsp), \old_rsp - .else - movq %rsp, \old_rsp - .endif - - .if \regs - UNWIND_HINT_REGS base=\old_rsp - .endif - - incl PER_CPU_VAR(irq_count) - jnz .Lirq_stack_push_old_rsp_\@ - - /* - * Right now, if we just incremented irq_count to zero, we've - * claimed the IRQ stack but we haven't switched to it yet. - * - * If anything is added that can interrupt us here without using IST, - * it must be *extremely* careful to limit its stack usage. This - * could include kprobes and a hypothetical future IST-less #DB - * handler. - * - * The OOPS unwinder relies on the word at the top of the IRQ - * stack linking back to the previous RSP for the entire time we're - * on the IRQ stack. For this to work reliably, we need to write - * it before we actually move ourselves to the IRQ stack. - */ - - movq \old_rsp, PER_CPU_VAR(irq_stack_backing_store + IRQ_STACK_SIZE - 8) - movq PER_CPU_VAR(hardirq_stack_ptr), %rsp - -#ifdef CONFIG_DEBUG_ENTRY - /* - * If the first movq above becomes wrong due to IRQ stack layout - * changes, the only way we'll notice is if we try to unwind right - * here. Assert that we set up the stack right to catch this type - * of bug quickly. - */ - cmpq -8(%rsp), \old_rsp - je .Lirq_stack_okay\@ - ud2 - .Lirq_stack_okay\@: -#endif - -.Lirq_stack_push_old_rsp_\@: - pushq \old_rsp - - .if \regs - UNWIND_HINT_REGS indirect=1 - .endif - - .if \save_ret - /* - * Push the return address to the stack. This return address can - * be found at the "real" original RSP, which was offset by 8 at - * the beginning of this macro. - */ - pushq -8(\old_rsp) - .endif -.endm - -/* - * Undoes ENTER_IRQ_STACK. - */ -.macro LEAVE_IRQ_STACK regs=1 - DEBUG_ENTRY_ASSERT_IRQS_OFF - /* We need to be off the IRQ stack before decrementing irq_count. */ - popq %rsp - - .if \regs - UNWIND_HINT_REGS - .endif - - /* - * As in ENTER_IRQ_STACK, irq_count == 0, we are still claiming - * the irq stack but we're not on it. - */ - - decl PER_CPU_VAR(irq_count) -.endm - /** * idtentry_body - Macro to emit code calling the C function * @cfunc: C function to be called
Re: [patch V6 32/37] x86/entry/64: Remove IRQ stack switching ASM
Quoted 1 line · Thomas Gleixner
PATCH v6 33/37 x86/entry: Make enter_from_user_mode() static
In reply to Thomas GleixnerThe ASM users are gone. All callers are local.
arch/x86/entry/common.c +1 -1
@@ -54,7 +54,7 @@* 2) Invoke context tracking if enabled to reactivate RCU * 3) Trace interrupts off state */ -__visible noinstr void enter_from_user_mode(void) +static noinstr void enter_from_user_mode(void) {enum ctx_state state =ct_state ();
Re: [patch V6 33/37] x86/entry: Make enter_from_user_mode() static
Quoted 3 lines · Thomas Gleixner
The ASM users are gone. All callers are local.
PATCH v6 34/37 x86/entry/32: Remove redundant irq disable code
In reply to Thomas GleixnerAll exceptions/interrupts return with interrupts disabled now. No point in doing this in ASM again.
--- arch/x86/entry/entry_32.S | 76 ---------------------------------------------- 1 file changed, 76 deletions(-)
arch/x86/entry/entry_32.S +0 -76
@@ -51,34 +51,6 @@ .section .entry.text, "ax" -/* - * We use macros for low-level operations which need to be overridden - * for paravirtualization. The following will never clobber any registers: - * INTERRUPT_RETURN (aka. "iret") - * GET_CR0_INTO_EAX (aka. "movl %cr0, %eax") - * ENABLE_INTERRUPTS_SYSEXIT (aka "sti; sysexit"). - * - * For DISABLE_INTERRUPTS/ENABLE_INTERRUPTS (aka "cli"/"sti"), you must - * specify what registers can be overwritten (CLBR_NONE, CLBR_EAX/EDX/ECX/ANY). - * Allowing a register to be clobbered can shrink the paravirt replacement - * enough to patch inline, increasing performance. - */ - -#ifdef CONFIG_PREEMPTION -# define preempt_stop(clobbers) DISABLE_INTERRUPTS(clobbers); TRACE_IRQS_OFF -#else -# define preempt_stop(clobbers) -#endif - -.macro TRACE_IRQS_IRET -#ifdef CONFIG_TRACE_IRQFLAGS - testl $X86_EFLAGS_IF, PT_EFLAGS(%esp) # interrupts off? - jz 1f - TRACE_IRQS_ON -1: -#endif -.endm - #define PTI_SWITCH_MASK (1 << PAGE_SHIFT) /*
@@ -881,38 +853,6 @@ SYM_CODE_START(ret_from_fork) SYM_CODE_END(ret_from_fork) .popsection -/* - * Return to user mode is not as complex as all this looks, - * but we want the default path for a system call return to - * go as quickly as possible which is why some of this is - * less clear than it otherwise should be. - */ - - # userspace resumption stub bypassing syscall exit tracing -SYM_CODE_START_LOCAL(ret_from_exception) - preempt_stop(CLBR_ANY) -ret_from_intr: -#ifdef CONFIG_VM86 - movl PT_EFLAGS(%esp), %eax # mix EFLAGS and CS - movb PT_CS(%esp), %al - andl $(X86_EFLAGS_VM | SEGMENT_RPL_MASK), %eax -#else - /* - * We can be coming here from child spawned by kernel_thread(). - */ - movl PT_CS(%esp), %eax - andl $SEGMENT_RPL_MASK, %eax -#endif - cmpl $USER_RPL, %eax - jb restore_all_kernel # not returning to v8086 or userspace - - DISABLE_INTERRUPTS(CLBR_ANY) - TRACE_IRQS_OFF - movl %esp, %eax - call prepare_exit_to_usermode - jmp restore_all_switch_stack -SYM_CODE_END(ret_from_exception) - SYM_ENTRY(__begin_SYSENTER_singlestep_region, SYM_L_GLOBAL, SYM_A_NONE) /* * All code from here through __end_SYSENTER_singlestep_region is subject
@@ -1147,22 +1087,6 @@ SYM_FUNC_START(entry_INT80_32) */ INTERRUPT_RETURN -restore_all_kernel: -#ifdef CONFIG_PREEMPTION - DISABLE_INTERRUPTS(CLBR_ANY) - cmpl $0, PER_CPU_VAR(__preempt_count) - jnz .Lno_preempt - testl $X86_EFLAGS_IF, PT_EFLAGS(%esp) # interrupts off (exception path) ? - jz .Lno_preempt - call preempt_schedule_irq -.Lno_preempt: -#endif - TRACE_IRQS_IRET - PARANOID_EXIT_TO_KERNEL_MODE - BUG_IF_WRONG_CR3 - RESTORE_REGS 4 - jmp .Lirq_return - .section .fixup, "ax" SYM_CODE_START(asm_iret_error) pushl $0 # no error code
Re: [patch V6 34/37] x86/entry/32: Remove redundant irq disable code
Quoted 4 lines · Thomas Gleixner
All exceptions/interrupts return with interrupts disabled now. No point in doing this in ASM again.
PATCH v6 35/37 x86/entry/64: Remove TRACE_IRQS_*_DEBUG
In reply to Thomas GleixnerSince INT3/#BP no longer runs on an IST, this workaround is no longer required.
Tested by running lockdep+ftrace as described in the initial commit:
5963e317b1e9 ("ftrace/x86: Do not change stacks in DEBUG when calling lockdep")
arch/x86/entry/entry_64.S +3 -45
@@ -68,44 +68,6 @@ SYM_CODE_END(native_usergs_sysret64) .endm /* - * When dynamic function tracer is enabled it will add a breakpoint - * to all locations that it is about to modify, sync CPUs, update - * all the code, sync CPUs, then remove the breakpoints. In this time - * if lockdep is enabled, it might jump back into the debug handler - * outside the updating of the IST protection. (TRACE_IRQS_ON/OFF). - * - * We need to change the IDT table before calling TRACE_IRQS_ON/OFF to - * make sure the stack pointer does not get reset back to the top - * of the debug stack, and instead just reuses the current stack. - */ -#if defined(CONFIG_DYNAMIC_FTRACE) && defined(CONFIG_TRACE_IRQFLAGS) - -.macro TRACE_IRQS_OFF_DEBUG - call debug_stack_set_zero - TRACE_IRQS_OFF - call debug_stack_reset -.endm - -.macro TRACE_IRQS_ON_DEBUG - call debug_stack_set_zero - TRACE_IRQS_ON - call debug_stack_reset -.endm - -.macro TRACE_IRQS_IRETQ_DEBUG - btl $9, EFLAGS(%rsp) /* interrupts off? */ - jnc 1f - TRACE_IRQS_ON_DEBUG -1: -.endm - -#else -# define TRACE_IRQS_OFF_DEBUG TRACE_IRQS_OFF -# define TRACE_IRQS_ON_DEBUG TRACE_IRQS_ON -# define TRACE_IRQS_IRETQ_DEBUG TRACE_IRQS_IRETQ -#endif - -/* * 64-bit SYSCALL instruction entry. Up to 6 arguments in registers. * * This is the only entry point used for 64-bit system calls. The
@@ -500,11 +462,7 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_REGS - .if \vector == X86_TRAP_DB - TRACE_IRQS_OFF_DEBUG - .else - TRACE_IRQS_OFF - .endif + TRACE_IRQS_OFF movq %rsp, %rdi /* pt_regs pointer */
@@ -924,7 +882,7 @@ SYM_CODE_END(paranoid_entry) SYM_CODE_START_LOCAL(paranoid_exit) UNWIND_HINT_REGS DISABLE_INTERRUPTS(CLBR_ANY) - TRACE_IRQS_OFF_DEBUG + TRACE_IRQS_OFF testl %ebx, %ebx /* swapgs needed? */ jnz .Lparanoid_exit_no_swapgs TRACE_IRQS_IRETQ
@@ -933,7 +891,7 @@ SYM_CODE_START_LOCAL(paranoid_exit) SWAPGS_UNSAFE_STACK jmp restore_regs_and_return_to_kernel .Lparanoid_exit_no_swapgs: - TRACE_IRQS_IRETQ_DEBUG + TRACE_IRQS_IRETQ /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%rbx save_reg=%r14 jmp restore_regs_and_return_to_kernel
Re: [patch V6 35/37] x86/entry/64: Remove TRACE_IRQS_*_DEBUG
Quoted 8 lines · Thomas Gleixner
Since INT3/#BP no longer runs on an IST, this workaround is no longer required.
Tested by running lockdep+ftrace as described in the initial commit:
5963e317b1e9("ftrace/x86: Do not change stacks in DEBUG when calling lockdep")
PATCH v6 36/37 x86/entry: Move paranoid irq tracing out of ASM code
In reply to Thomas Gleixner--- arch/x86/entry/entry_64.S | 13 ------------- arch/x86/kernel/nmi.c | 3 +++ 2 files changed, 3 insertions(+), 13 deletions(-)
arch/x86/entry/entry_64.S +0 -13
@@ -16,7 +16,6 @@ * * Some macro usage: * - SYM_FUNC_START/END:Define functions in the symbol table. - * - TRACE_IRQ_*: Trace hardirq state for lock debugging. * - idtentry: Define exception entry points. */ #include <linux/linkage.h>
@@ -107,11 +106,6 @@ SYM_CODE_END(native_usergs_sysret64) SYM_CODE_START(entry_SYSCALL_64) UNWIND_HINT_EMPTY - /* - * Interrupts are off on entry. - * We do not frame this tiny irq-off block with TRACE_IRQS_OFF/ON, - * it is too small to ever cause noticeable irq latency. - */ swapgs /* tss.sp2 is scratch space. */
@@ -462,8 +456,6 @@ SYM_CODE_START(\asmsym) UNWIND_HINT_REGS - TRACE_IRQS_OFF - movq %rsp, %rdi /* pt_regs pointer */ .if \vector == X86_TRAP_DB
@@ -881,17 +873,13 @@ SYM_CODE_END(paranoid_entry) */ SYM_CODE_START_LOCAL(paranoid_exit) UNWIND_HINT_REGS - DISABLE_INTERRUPTS(CLBR_ANY) - TRACE_IRQS_OFF testl %ebx, %ebx /* swapgs needed? */ jnz .Lparanoid_exit_no_swapgs - TRACE_IRQS_IRETQ /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%rbx save_reg=%r14 SWAPGS_UNSAFE_STACK jmp restore_regs_and_return_to_kernel .Lparanoid_exit_no_swapgs: - TRACE_IRQS_IRETQ /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%rbx save_reg=%r14 jmp restore_regs_and_return_to_kernel
@@ -1292,7 +1280,6 @@ SYM_CODE_START(asm_exc_nmi) call paranoid_entry UNWIND_HINT_REGS - /* paranoidentry exc_nmi(), 0; without TRACE_IRQS_OFF */ movq %rsp, %rdi movq $-1, %rsi call exc_nmi
arch/x86/kernel/nmi.c +3 -0
@@ -334,6 +334,7 @@ static noinstr void default_do_nmi(struc__this_cpu_write (last_nmi_rip, regs->ip);instrumentation_begin (); +trace_hardirqs_off_prepare ();ftrace_nmi_handler_enter (); handled =nmi_handle (NMI_LOCAL , regs);
@@ -422,6 +423,8 @@ static noinstr void default_do_nmi(struc out:ftrace_nmi_handler_exit (); +if (regs->flags &X86_EFLAGS_IF ) +trace_hardirqs_on_prepare ();instrumentation_end (); }
Andy Lutomirski
maintainerRe: [patch V6 36/37] x86/entry: Move paranoid irq tracing out of ASM code
Quoted 1 line · Thomas Gleixner
I think something's missing here. With this patch applied, don't we get to exc_debug_kernel() -> handle_debug() without doing idtentry_enter() or equivalent? And that can even enable IRQs.
Maybe exc_debug_kernel() should wrap handle_debug() in some appropriate _enter() / _exit() pair?
--Andy
Re: [patch V6 36/37] x86/entry: Move paranoid irq tracing out of ASM code
Quoted 8 lines · Andy Lutomirski
On Fri, May 15, 2020 at 5:11 PM Thomas Gleixner <[email protected]> wrote:
I think something's missing here. With this patch applied, don't we get to
exc_debug_kernel()->handle_debug()without doingidtentry_enter()or equivalent? And that can even enable IRQs.Maybe
exc_debug_kernel()should wraphandle_debug()in some appropriate_enter()/_exit()pair?
I'm the one who is missing something here, i.e. the connection of this patch to #DB. exc_debug_kernel() still looks like this:
nmi_enter_notrace(); handle_debug(regs, dr6, false); nmi_exit_notrace();
Confused.
Andy Lutomirski
maintainerRe: [patch V6 36/37] x86/entry: Move paranoid irq tracing out of ASM code
Quoted 12 lines · Thomas Gleixner
Quoted 8 lines · Andy Lutomirski
On Fri, May 15, 2020 at 5:11 PM Thomas Gleixner <[email protected]> wrote:
I think something's missing here. With this patch applied, don't we get to
exc_debug_kernel()->handle_debug()without doingidtentry_enter()or equivalent? And that can even enable IRQs.Maybe
exc_debug_kernel()should wraphandle_debug()in some appropriate_enter()/_exit()pair?
Quoted 9 lines
I'm the one who is missing something here, i.e. the connection of this patch to #DB.
exc_debug_kernel()still looks like this:nmi_enter_notrace(); handle_debug(regs, dr6, false); nmi_exit_notrace();Confused.
Hmm. I guess the code is correct-ish or at least as correct as it ever was. But $SUBJECT says "Move paranoid irq tracing out of ASM code" but you didn't move it into all the users. So now the NMI code does trace_hardirqs_on_prepare() but the #DB code doesn't. Perhaps the changelog should mention this.
exc_kernel_debug() is an atrocity. Every now and then I get started on cleanup it up and so far I always get mired in the giant amount of indirection.
So Acked-by: Andy Lutomirski <[email protected]> if you write a credible changelog.
Re: [patch V6 36/37] x86/entry: Move paranoid irq tracing out of ASM code
Quoted 20 lines · Andy Lutomirski
Quoted 19 lines · Thomas Gleixner
Quoted 8 lines · Andy Lutomirski
On Fri, May 15, 2020 at 5:11 PM Thomas Gleixner <[email protected]> wrote:
I think something's missing here. With this patch applied, don't we get to
exc_debug_kernel()->handle_debug()without doingidtentry_enter()or equivalent? And that can even enable IRQs.Maybe
exc_debug_kernel()should wraphandle_debug()in some appropriate_enter()/_exit()pair?I'm the one who is missing something here, i.e. the connection of this patch to #DB.
exc_debug_kernel()still looks like this:nmi_enter_notrace(); handle_debug(regs, dr6, false); nmi_exit_notrace();
Quoted 8 lines
Quoted 2 lines
Confused.
Hmm. I guess the code is correct-ish or at least as correct as it ever was. But $SUBJECT says "Move paranoid irq tracing out of ASM code" but you didn't move it into all the users. So now the NMI code does
trace_hardirqs_on_prepare()but the #DB code doesn't. Perhaps the changelog should mention this.
Duh. I simply missed to add it.
Quoted 6 lines
exc_kernel_debug()is an atrocity. Every now and then I get started on cleanup it up and so far I always get mired in the giant amount of indirection.So Acked-by: Andy Lutomirski <[email protected]> if you write a credible changelog.
PATCH v6 37/37 x86/entry: Remove the TRACE_IRQS cruft
In reply to Thomas GleixnerNo more users.
arch/x86/entry/entry_64.S +0 -13
@@ -53,19 +53,6 @@ SYM_CODE_START(native_usergs_sysret64) SYM_CODE_END(native_usergs_sysret64) #endif /* CONFIG_PARAVIRT */ -.macro TRACE_IRQS_FLAGS flags:req -#ifdef CONFIG_TRACE_IRQFLAGS - btl $9, \flags /* interrupts off? */ - jnc 1f - TRACE_IRQS_ON -1: -#endif -.endm - -.macro TRACE_IRQS_IRETQ - TRACE_IRQS_FLAGS EFLAGS(%rsp) -.endm - /* * 64-bit SYSCALL instruction entry. Up to 6 arguments in registers. *
arch/x86/entry/thunk_64.S +1 -8
@@ -3,7 +3,6 @@ * Save registers before calling assembly functions. This avoids * disturbance of register allocation in some inline assembly constructs. * Copyright 2001,2002 by Andi Kleen, SuSE Labs. - * Added trace_hardirqs callers - Copyright 2007 Steven Rostedt, Red Hat, Inc. */ #include <linux/linkage.h> #include "calling.h"
@@ -37,11 +36,6 @@ SYM_FUNC_END(\name) _ASM_NOKPROBE(\name) .endm -#ifdef CONFIG_TRACE_IRQFLAGS - THUNK trace_hardirqs_on_thunk,trace_hardirqs_on_caller,1 - THUNK trace_hardirqs_off_thunk,trace_hardirqs_off_caller,1 -#endif - #ifdef CONFIG_PREEMPTION THUNK preempt_schedule_thunk, preempt_schedule THUNK preempt_schedule_notrace_thunk, preempt_schedule_notrace
@@ -49,8 +43,7 @@ SYM_FUNC_END(\name) EXPORT_SYMBOL(preempt_schedule_notrace_thunk) #endif -#if defined(CONFIG_TRACE_IRQFLAGS) \ - || defined(CONFIG_PREEMPTION) +#ifdef CONFIG_PREEMPTION SYM_CODE_START_LOCAL_NOALIGN(.L_restore) popq %r11 popq %r10
arch/x86/include/asm/irqflags.h +0 -10
@@ -172,14 +172,4 @@ static inline int arch_irqs_disabled(void) }#endif /* !__ASSEMBLY__ */ -#ifdef __ASSEMBLY__ -#ifdef CONFIG_TRACE_IRQFLAGS -# define TRACE_IRQS_ON call trace_hardirqs_on_thunk; -# define TRACE_IRQS_OFF call trace_hardirqs_off_thunk; -#else -# define TRACE_IRQS_ON -# define TRACE_IRQS_OFF -#endif -#endif /* __ASSEMBLY__ */ -#endif
Re: [patch V6 37/37] x86/entry: Remove the TRACE_IRQS cruft
Quoted 3 lines · Thomas Gleixner
No more users.
Hallelujah!
Quoted 1 line
Paul E. McKenney
maintainerOn Sat, May 16, 2020 at 01:45:47AM +0200, Thomas Gleixner wrote:
[ . . . ]
Quoted 8 lines
- noinstr-rcu-nmi-2020-05-15
Based on the core/rcu branch in the tip tree. It has merged in noinstr-lds-2020-05-15 and contains the nmi_enter/exit() changes along with the noinstr section changes on top.
This tag is intended to be pulled by Paul into his rcu/next branch so he can sort the conflicts and base further work on top.
And this sorting process is now allegedly complete and available on the -rcu tree's "dev" branch. As you might have guessed, the major source of conflicts were with Joel's patches, including one conflict that was invisible to "git rebase":
1b2530e7d0c3 ("Revert b8c17e6664c4 ("rcu: Maintain special bits at bottom of ->dynticks counter") 03f31532d0ce ("rcu/tree: Add better tracing for dyntick-idle")a309d5ce2335 ("rcu/tree: Clean up dynticks counter usage")5c6e734fbaeb ("rcu/tree: Remove dynticks_nmi_nesting counter")
This passes modest rcutorture testing. So far, so good! ;-)
Thanx, Paul
Quoted 18 lines · Paul E. McKenney
On Sat, May 16, 2020 at 01:45:47AM +0200, Thomas Gleixner wrote:
[ . . . ]
Quoted 8 lines
- noinstr-rcu-nmi-2020-05-15
Based on the core/rcu branch in the tip tree. It has merged in noinstr-lds-2020-05-15 and contains the nmi_enter/exit() changes along with the noinstr section changes on top.
This tag is intended to be pulled by Paul into his rcu/next branch so he can sort the conflicts and base further work on top.
And this sorting process is now allegedly complete and available on the -rcu tree's "dev" branch. As you might have guessed, the major source of conflicts were with Joel's patches, including one conflict that was invisible to "git rebase":
Quoted 6 lines
1b2530e7d0c3("Revertb8c17e6664c4("rcu: Maintain special bits at bottom of ->dynticks counter")03f31532d0ce("rcu/tree: Add better tracing for dyntick-idle")a309d5ce2335("rcu/tree: Clean up dynticks counter usage")5c6e734fbaeb("rcu/tree: Removedynticks_nmi_nestingcounter")This passes modest rcutorture testing. So far, so good! ;-)
Also I double checked these patches in the rcu/dev branch. It looks good to me, thanks for resolving the conflicts!!
- Joel
Peter Zijlstra
maintainerSo on top of you entry-v8-full; I had to chase one of those instrumentation_end() escapes an (extended) basic block chase (again!).
How about we do something like the below; that fixes the current case (rcu_eqs_enter) but also kills the entire class.
--- arch/x86/include/asm/bug.h | 2 +- include/linux/compiler.h | 16 +++++++++++++--- include/linux/compiler_types.h | 4 ---- 3 files changed, 14 insertions(+), 8 deletions(-)
arch/x86/include/asm/bug.h +1 -1
@@ -79,8 +79,8 @@ do { \do { \instrumentation_begin (); \_BUG_FLAGS (ASM_UD2 ,BUGFLAG_WARNING |(flags)); \ -instrumentation_end (); \annotate_reachable (); \ +instrumentation_end (); \ } while (0)#include <asm-generic/bug.h>
include/linux/compiler.h +13 -3
@@ -120,25 +120,35 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,/* Annotate a C jump table to allow objtool to follow the code flow */ #define __annotate_jump_table __section(.rodata..c_jump_table) +#ifdef CONFIG_DEBUG_ENTRY +/* Section for code which can't be instrumented at all */ +#define noinstr \ + noinline notrace __attribute((__section__(".noinstr.text"))) +/* Begin/end of an instrumentation safe region */ -#define instrumentation_begin () ({ \ +#define instrumentation_begin () ({ \ asm volatile("%c0:\n\t" \ ".pushsection .discard.instr_begin\n\t" \ ".long %c0b - .\n\t" \ ".popsection\n\t" : : "i" (__COUNTER__)); \ }) -#define instrumentation_end () ({ \ - asm volatile("%c0:\n\t" \ +#define instrumentation_end () ({ \ + asm volatile("%c0: nop\n\t" \ ".pushsection .discard.instr_end\n\t" \ ".long %c0b - .\n\t" \ ".popsection\n\t" : : "i" (__COUNTER__)); \ }) +#endif /* CONFIG_DEBUG_ENTRY */ #else #define annotate_reachable ()#define annotate_unreachable ()#define __annotate_jump_table +#endif + +#ifndef noinstr +#define noinstr noinline notrace#define instrumentation_begin () do { } while(0)#define instrumentation_end () do { } while(0)#endif
include/linux/compiler_types.h +0 -4
@@ -118,10 +118,6 @@ struct ftrace_likely_data {#define notrace __attribute__((__no_instrument_function__))#endif -/* Section for code which can't be instrumented at all */ -#define noinstr \ - noinline notrace __attribute((__section__(".noinstr.text"))) -/* * it doesn't make sense on ARM (currently the only user of __naked) * to trace naked functions because then mcount is called without
Quoted 4 lines · Peter Zijlstra
So on top of you entry-v8-full; I had to chase one of those
instrumentation_end()escapes an (extended) basic block chase (again!).+#ifdef
CONFIG_DEBUG_ENTRY
Why this? We lose the kprobes runtime protection that way.
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val,
Quoted 11 lines
include/linux/compiler.h +5
-1+/* Section for code which can't be instrumented at all */ +#define noinstr \ + noinline notrace __attribute((__section__(".noinstr.text"))) +/* Begin/end of an instrumentation safe region */ -#define instrumentation_begin () ({ \ +#define instrumentation_begin () ({ \ asm volatile("%c0:\n\t" \ ".pushsection .discard.instr_begin\n\t" \ ".long %c0b - .\n\t" \ ".popsection\n\t" : : "i" (__COUNTER__));
Nifty.
Thanks,
tglx
Peter Zijlstra
maintainerQuoted 7 lines · Thomas Gleixner
Quoted 4 lines · Peter Zijlstra
So on top of you entry-v8-full; I had to chase one of those
instrumentation_end()escapes an (extended) basic block chase (again!).+#ifdef
CONFIG_DEBUG_ENTRYWhy this? We lose the kprobes runtime protection that way.
Oh bugger indeed. I forgot about that :-(
I added the CONFIG_DEBUG_ENTRY dependency to instrumentation_{begin,end}() because they now emit actual code, and I figured we shouldn't bother 'production' kernels with all them extra NOPs.
And then I figured (wrongly!) that since I have that, I might as well add noinstr to is.
Quoted 5 lines
Quoted 5 lines
include/linux/compiler.h +4
-0+/* Section for code which can't be instrumented at all */ +#define noinstr \ + noinline notrace __attribute((__section__(".noinstr.text"))) +/* Begin/end of an instrumentation safe region */
Quoted 8 lines
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val,
Quoted 6 lines
include/linux/compiler.h +1
-1-#define instrumentation_begin () ({ \ +#define instrumentation_begin () ({ \ asm volatile("%c0:\n\t" \ ".pushsection .discard.instr_begin\n\t" \ ".long %c0b - .\n\t" \ ".popsection\n\t" : : "i" (__COUNTER__));Nifty.
Yeah, took a bit of fiddling because objtool is a bit weird vs UD2, but if you order it just right in the WARN thing it works :-)
You want a new delta without the noinstr thing on?
arch/x86/include/asm/bug.h do { \ · line 84
Quoted 13 lines · Peter Zijlstra
So on top of you entry-v8-full; I had to chase one of those
instrumentation_end()escapes an (extended) basic block chase (again!).arch/x86/include/asm/bug.h +1
-1@@ -79,8 +79,8 @@ do { \do { \instrumentation_begin (); \_BUG_FLAGS (ASM_UD2 ,BUGFLAG_WARNING |(flags)); \ -instrumentation_end (); \annotate_reachable (); \ +instrumentation_end (); \ }while (0 )
I just applied this part and rebuilt:
vmlinux.o: warning: objtool: rcu_eqs_enter.constprop.77()+0xa9: call to rcu_preempt_deferred_qs() leaves .noinstr.text section
Did it go away after you disabled DEBUG_ENTRY perhaps?
Peter Zijlstra
maintainerQuoted 17 lines · Thomas Gleixner
arch/x86/include/asm/bug.h do { \ · line 84
Quoted 13 lines · Peter Zijlstra
So on top of you entry-v8-full; I had to chase one of those
instrumentation_end()escapes an (extended) basic block chase (again!).arch/x86/include/asm/bug.h +1
-1@@ -79,8 +79,8 @@ do { \do { \instrumentation_begin (); \_BUG_FLAGS (ASM_UD2 ,BUGFLAG_WARNING |(flags)); \ -instrumentation_end (); \annotate_reachable (); \ +instrumentation_end (); \ }while (0 )I just applied this part and rebuilt:
Quoted 4 lines
vmlinux.o: warning: objtool: rcu_eqs_enter.constprop.77()+0xa9: call to
rcu_preempt_deferred_qs()leaves .noinstr.text sectionDid it go away after you disabled
DEBUG_ENTRYperhaps?
Hehe, then all complaints would be gone :-)
So tglx/entry-v8-full + below patch:
$ make O=defconfig-build clean ... $ make CC=gcc-9 O=defconfig-build/ vmlinux -j40 -s vmlinux.o: warning: objtool: exc_debug()+0x158: call to trace_hwlat_timestamp() leaves .noinstr.text section vmlinux.o: warning: objtool: exc_nmi()+0x190: call to trace_hwlat_timestamp() leaves .noinstr.text section vmlinux.o: warning: objtool: do_machine_check()+0x46: call to mce_rdmsrl() leaves .noinstr.text section $
(it really isn't defconfig, but your config-fail + DEBUG_ENTRY)
--- arch/x86/include/asm/bug.h | 2 +- include/linux/compiler.h | 11 ++++++++--- 2 files changed, 9 insertions(+), 4 deletions(-)
arch/x86/include/asm/bug.h +1 -1
@@ -79,8 +79,8 @@ do { \do { \instrumentation_begin (); \_BUG_FLAGS (ASM_UD2 ,BUGFLAG_WARNING |(flags)); \ -instrumentation_end (); \annotate_reachable (); \ +instrumentation_end (); \ } while (0)#include <asm-generic/bug.h>
include/linux/compiler.h +8 -3
@@ -120,25 +120,30 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,/* Annotate a C jump table to allow objtool to follow the code flow */ #define __annotate_jump_table __section(.rodata..c_jump_table) +#ifdef CONFIG_DEBUG_ENTRY /* Begin/end of an instrumentation safe region */ -#define instrumentation_begin () ({ \ +#define instrumentation_begin () ({ \ asm volatile("%c0:\n\t" \ ".pushsection .discard.instr_begin\n\t" \ ".long %c0b - .\n\t" \ ".popsection\n\t" : : "i" (__COUNTER__)); \ }) -#define instrumentation_end () ({ \ - asm volatile("%c0:\n\t" \ +#define instrumentation_end () ({ \ + asm volatile("%c0: nop\n\t" \ ".pushsection .discard.instr_end\n\t" \ ".long %c0b - .\n\t" \ ".popsection\n\t" : : "i" (__COUNTER__)); \ }) +#endif /* CONFIG_DEBUG_ENTRY */ #else #define annotate_reachable ()#define annotate_unreachable ()#define __annotate_jump_table +#endif + +#ifndef instrumentation_begin#define instrumentation_begin () do { } while(0)#define instrumentation_end () do { } while(0)#endif
Peter Zijlstra
maintainerQuoted 8 lines · Peter Zijlstra
$ make CC=gcc-9 O=defconfig-build/ vmlinux -j40 -s vmlinux.o: warning: objtool: exc_debug()+0x158: call to trace_hwlat_timestamp() leaves .noinstr.text section vmlinux.o: warning: objtool: exc_nmi()+0x190: call to trace_hwlat_timestamp() leaves .noinstr.text section vmlinux.o: warning: objtool: do_machine_check()+0x46: call to mce_rdmsrl() leaves .noinstr.text section $(it really isn't defconfig, but your config-fail +
DEBUG_ENTRY)
With comment on, as requested.
---
arch/x86/include/asm/bug.h +1 -1
@@ -79,8 +79,8 @@ do { \do { \instrumentation_begin (); \_BUG_FLAGS (ASM_UD2 ,BUGFLAG_WARNING |(flags)); \ -instrumentation_end (); \annotate_reachable (); \ +instrumentation_end (); \ } while (0)#include <asm-generic/bug.h>
include/linux/compiler.h +39 -3
64 lines not on this page · raw message
Quoted 39 lines · Peter Zijlstra
include/linux/compiler.h +29
-3+#ifdef CONFIG_DEBUG_ENTRY /* Begin/end of an instrumentation safe region */ -#define instrumentation_begin () ({ \ +#define instrumentation_begin () ({ \ asm volatile("%c0:\n\t" \ ".pushsection .discard.instr_begin\n\t" \ ".long %c0b - .\n\t" \ ".popsection\n\t" : : "i" (__COUNTER__)); \ }) -#define instrumentation_end () ({ \ - asm volatile("%c0:\n\t" \ +/* +* Because instrumentation_{begin,end}() can nest, objtool validation considers +* _begin() a +1 and _end() a -1 and computes a sum over the instructions. +* When the value is greater than 0, we consider instrumentation allowed. +* +* There is a problem with code like: +* +* noinstr void foo() +* { +* instrumentation_begin(); +* ... +* if (cond) { +* instrumentation_begin(); +* ... +* instrumentation_end(); +* } +* bar(); +* instrumentation_end(); +* } +* +* If instrumentation_end() would be an empty label, like all the other +* annotations, the inner _end(), which is at the end of a conditional block, +* would land on the instruction after the block. +* +* If we then consider the sum of the !cond path, we'll see that the call to +* bar() is with a 0-value, even though, we meant it to happen with a positive +* value.
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 168
Quoted 10 lines
include/linux/compiler.h +6
-0+* +* To avoid this, have _end() be a NOP instruction, this ensures it will be +* part of the condition block and does not escape. +*/ +#define instrumentation_end () ({ \ + asm volatile("%c0: nop\n\t" \ ".pushsection .discard.instr_end\n\t" \ ".long %c0b - .\n\t" \ ".popsection\n\t" : : "i" (__COUNTER__)); \ })
Hello,
I, who don't know how does the objtool handle it, am just curious. _begin() and _end() are symmetrical, which means if _end() (without nop) can escape, so can _begin() in a reverse way. For example:
noinstr void foo()
{
instrumentation_begin();
do {
instrumentation_begin();
...
instrumentation_end();
} while (cond);
bar();
instrumentation_end();
}Here, the first _begin() can be "dragged" into the do-while block. Expectedly, objtool validation should not complain here.
But objtool validation's not complaining means it can handle it magically correctly (by distinguishing how many _begin()s should be taken around the jmp target when jmp in a specific path), or handle it by not checking if all paths have the same count onto a jmp target (a little nervous to me), or other possible ways.
Sorry for my curiosity.
Thanks
Lai.
Peter Zijlstra
maintainerQuoted 18 lines · Lai Jiangshan
Hello,
I, who don't know how does the objtool handle it, am just curious.
_begin()and_end()are symmetrical, which means if_end()(without nop) can escape, so can_begin()in a reverse way. For example:noinstr void
foo(){ instrumentation_begin(); do { instrumentation_begin(); ... instrumentation_end(); } while (cond); bar(); instrumentation_end(); }
Quoted 8 lines
Here, the first
_begin()can be "dragged" into the do-while block. Expectedly, objtool validation should not complain here.But objtool validation's not complaining means it can handle it magically correctly (by distinguishing how many _begin()s should be taken around the jmp target when jmp in a specific path), or handle it by not checking if all paths have the same count onto a jmp target (a little nervous to me), or other possible ways.
No, I tihnk you're right. It could be we never hit this particular problem. Even the one described, where end leaks out, is quite rare. For instance, the last one I debgged (that led to this patch) only showed itself with gcc-9, but not with gcc-8 for example.
Anyway, if we ever find the above, I'll add the NOP to begin too.
Peter Zijlstra
maintainerQuoted 29 lines · Peter Zijlstra
Quoted 18 lines · Lai Jiangshan
Hello,
I, who don't know how does the objtool handle it, am just curious.
_begin()and_end()are symmetrical, which means if_end()(without nop) can escape, so can_begin()in a reverse way. For example:noinstr void
foo(){ instrumentation_begin(); do { instrumentation_begin(); ... instrumentation_end(); } while (cond); bar(); instrumentation_end(); }Quoted 8 lines
Here, the first
_begin()can be "dragged" into the do-while block. Expectedly, objtool validation should not complain here.But objtool validation's not complaining means it can handle it magically correctly (by distinguishing how many _begin()s should be taken around the jmp target when jmp in a specific path), or handle it by not checking if all paths have the same count onto a jmp target (a little nervous to me), or other possible ways.
Quoted 6 lines
No, I tihnk you're right. It could be we never hit this particular problem. Even the one described, where end leaks out, is quite rare. For instance, the last one I debgged (that led to this patch) only showed itself with gcc-9, but not with gcc-8 for example.
Anyway, if we ever find the above, I'll add the NOP to begin too.
FYI, I just found one, I'll be making instrumentation_begin() a NOP too.
Quoted 17 lines · Peter Zijlstra
On Mon, May 18, 2020 at 10:24:53PM +0200, Thomas Gleixner wrote:
So tglx/entry-v8-full + below patch:$ make O=defconfig-build clean ... $ make CC=gcc-9 O=defconfig-build/ vmlinux -j40 -s vmlinux.o: warning: objtool: exc_debug()+0x158: call to trace_hwlat_timestamp() leaves .noinstr.text section vmlinux.o: warning: objtool: exc_nmi()+0x190: call to trace_hwlat_timestamp() leaves .noinstr.text section vmlinux.o: warning: objtool: do_machine_check()+0x46: call to mce_rdmsrl() leaves .noinstr.text section $(it really isn't defconfig, but your config-fail +
DEBUG_ENTRY)include/linux/compiler.h +2
-1+#ifdef CONFIG_DEBUG_ENTRY /* Begin/end of an instrumentation safe region */ -#define instrumentation_begin () ({ \ +#define instrumentation_begin () ({ \
include/linux/compiler.h void ftrace_likely_update(struct ftrace_likely_data *f, int val, · line 133
Quoted 10 lines
include/linux/compiler.h +2
-2asmvolatile ("%c0:\n\t" \".pushsection .discard.instr_begin\n\t" \".long %c0b - .\n\t" \".popsection\n\t" : :"i" (__COUNTER__)); \ }) -#define instrumentation_end () ({ \ - asm volatile("%c0:\n\t" \ +#define instrumentation_end () ({ \ + asm volatile("%c0: nop\n\t" \
Bah. I fatfingered that nop out when I fixed up that noinstr wreckage. With that added back it does was it claims to do.
Steven Rostedt
maintainerQuoted 3 lines · On Sat, 16 May 2020 01:45:47 +0200 Thomas Gleixner <[email protected]> wrote:
The V6 leftover series is based on:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.gitentry-base-v6
$ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-base-v6 fatal: couldn't find remote ref entry-base-v6
-- Steve
Quoted 10 lines · Steven Rostedt
Quoted 3 lines · On Sat, 16 May 2020 01:45:47 +0200 Thomas Gleixner <[email protected]> wrote:
The V6 leftover series is based on:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.gitentry-base-v6$ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-base-v6 fatal: couldn't find remote ref entry-base-v6
try entry-v6-base or better entry-v8-base
Steven Rostedt
maintainerQuoted 7 lines · On Tue, 19 May 2020 21:09:48 +0200 Thomas Gleixner <[email protected]> wrote:
Quoted 5 lines · Steven Rostedt
Quoted 2 lines · On Sat, 16 May 2020 01:45:47 +0200 Thomas Gleixner <[email protected]> wrote:
The V6 leftover series is based on:
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.gitentry-base-v6$ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git entry-base-v6 fatal: couldn't find remote ref entry-base-v6try entry-v6-base or better entry-v8-base
Ah, I should have had a V8!
-- Steve