cpuidle,rcu: Clean up the mess
Subsystems (66)
Peter Zijlstra
maintainerv3 cover cpuidle,rcu: Clean up the mess
Hi All!
The (hopefully) final respin of cpuidle vs rcu cleanup patches. Barring any objections I'll be queueing these patches in tip/sched/core in the next few days.
v2: https://lkml.kernel.org/r/[email protected]
These here patches clean up the mess that is cpuidle vs rcuidle.
At the end of the ride there's only on RCU_NONIDLE user left:
arch/arm64/kernel/suspend.c: RCU_NONIDLE(__cpu_suspend_exit());
And I know Mark has been prodding that with something sharp.
The last version was tested by a number of people and I'm hoping to not have broken anything in the meantime ;-)
Changes since v2:
- rebased to v6.2-rc3; as available at:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/idle
- folded: https://lkml.kernel.org/r/[email protected]
which makes the ARM cpuidle index 0 consistently not use CPUIDLE_FLAG_RCU_IDLE, as requested by Ulf.
- added a few more __always_inline to empty stub functions as found by the robot.
- Used _RET_IP_ instead of _THIS_IP_ in a few placed because of:
https://github.com/ClangBuiltLinux/linux/issues/263
- Added new patches to address various robot reports:
#35: trace,hardirq: No moar _rcuidle() tracing
#47: cpuidle: Ensure ct_cpuidle_enter() is always called from noinstr/__cpuidle
#48: cpuidle,arch: Mark all ct_cpuidle_enter() callers __cpuidle
#49: cpuidle,arch: Mark all regular cpuidle_state::enter methods __cpuidle
#50: cpuidle: Comments about noinstr/__cpuidle
#51: context_tracking: Fix noinstr vs KASAN--- arch/alpha/kernel/process.c | 1 - arch/alpha/kernel/vmlinux.lds.S | 1 - arch/arc/kernel/process.c | 3 ++ arch/arc/kernel/vmlinux.lds.S | 1 - arch/arm/include/asm/vmlinux.lds.h | 1 - arch/arm/kernel/cpuidle.c | 4 +- arch/arm/kernel/process.c | 1 - arch/arm/kernel/smp.c | 6 +-- arch/arm/mach-davinci/cpuidle.c | 4 +- arch/arm/mach-gemini/board-dt.c | 3 +- arch/arm/mach-imx/cpuidle-imx5.c | 4 +- arch/arm/mach-imx/cpuidle-imx6q.c | 8 ++-- arch/arm/mach-imx/cpuidle-imx6sl.c | 4 +- arch/arm/mach-imx/cpuidle-imx6sx.c | 9 ++-- arch/arm/mach-imx/cpuidle-imx7ulp.c | 4 +- arch/arm/mach-omap2/common.h | 6 ++- arch/arm/mach-omap2/cpuidle34xx.c | 16 ++++++- arch/arm/mach-omap2/cpuidle44xx.c | 29 +++++++------ arch/arm/mach-omap2/omap-mpuss-lowpower.c | 12 +++++- arch/arm/mach-omap2/pm.h | 2 +- arch/arm/mach-omap2/pm24xx.c | 51 +--------------------- arch/arm/mach-omap2/pm34xx.c | 14 +++++-- arch/arm/mach-omap2/pm44xx.c | 2 +- arch/arm/mach-omap2/powerdomain.c | 10 ++--- arch/arm/mach-s3c/cpuidle-s3c64xx.c | 5 +-- arch/arm64/kernel/cpuidle.c | 2 +- arch/arm64/kernel/idle.c | 1 - arch/arm64/kernel/smp.c | 4 +- arch/arm64/kernel/vmlinux.lds.S | 1 - arch/csky/kernel/process.c | 1 - arch/csky/kernel/smp.c | 2 +- arch/csky/kernel/vmlinux.lds.S | 1 - arch/hexagon/kernel/process.c | 1 - arch/hexagon/kernel/vmlinux.lds.S | 1 - arch/ia64/kernel/process.c | 1 + arch/ia64/kernel/vmlinux.lds.S | 1 - arch/loongarch/kernel/idle.c | 1 + arch/loongarch/kernel/vmlinux.lds.S | 1 - arch/m68k/kernel/vmlinux-nommu.lds | 1 - arch/m68k/kernel/vmlinux-std.lds | 1 - arch/m68k/kernel/vmlinux-sun3.lds | 1 - arch/microblaze/kernel/process.c | 1 - arch/microblaze/kernel/vmlinux.lds.S | 1 - arch/mips/kernel/idle.c | 14 +++---- arch/mips/kernel/vmlinux.lds.S | 1 - arch/nios2/kernel/process.c | 1 - arch/nios2/kernel/vmlinux.lds.S | 1 - arch/openrisc/kernel/process.c | 1 + arch/openrisc/kernel/vmlinux.lds.S | 1 - arch/parisc/kernel/process.c | 2 - arch/parisc/kernel/vmlinux.lds.S | 1 - arch/powerpc/kernel/idle.c | 5 +-- arch/powerpc/kernel/vmlinux.lds.S | 1 - arch/riscv/kernel/process.c | 1 - arch/riscv/kernel/vmlinux-xip.lds.S | 1 - arch/riscv/kernel/vmlinux.lds.S | 1 - arch/s390/kernel/idle.c | 1 - arch/s390/kernel/vmlinux.lds.S | 1 - arch/sh/kernel/idle.c | 1 + arch/sh/kernel/vmlinux.lds.S | 1 - arch/sparc/kernel/leon_pmc.c | 4 ++ arch/sparc/kernel/process_32.c | 1 - arch/sparc/kernel/process_64.c | 3 +- arch/sparc/kernel/vmlinux.lds.S | 1 - arch/um/kernel/dyn.lds.S | 1 - arch/um/kernel/process.c | 1 - arch/um/kernel/uml.lds.S | 1 - arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/coco/tdx/tdcall.S | 15 +------ arch/x86/coco/tdx/tdx.c | 25 ++++------- arch/x86/events/amd/brs.c | 13 +++--- arch/x86/include/asm/fpu/xcr.h | 4 +- arch/x86/include/asm/irqflags.h | 11 ++--- arch/x86/include/asm/mwait.h | 14 +++---- arch/x86/include/asm/nospec-branch.h | 2 +- arch/x86/include/asm/paravirt.h | 6 ++- arch/x86/include/asm/perf_event.h | 2 +- arch/x86/include/asm/shared/io.h | 4 +- arch/x86/include/asm/shared/tdx.h | 1 - arch/x86/include/asm/special_insns.h | 8 ++-- arch/x86/include/asm/xen/hypercall.h | 2 +- arch/x86/kernel/cpu/bugs.c | 2 +- arch/x86/kernel/fpu/core.c | 4 +- arch/x86/kernel/paravirt.c | 14 ++++++- arch/x86/kernel/process.c | 65 ++++++++++++++-------------- arch/x86/kernel/vmlinux.lds.S | 1 - arch/x86/lib/memcpy_64.S | 5 +-- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 4 +- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/irq.c | 2 +- arch/xtensa/kernel/process.c | 1 + arch/xtensa/kernel/vmlinux.lds.S | 1 - drivers/acpi/processor_idle.c | 28 ++++++++----- drivers/base/power/runtime.c | 24 +++++------ drivers/clk/clk.c | 8 ++-- drivers/cpuidle/cpuidle-arm.c | 4 +- drivers/cpuidle/cpuidle-big_little.c | 12 ++++-- drivers/cpuidle/cpuidle-mvebu-v7.c | 13 ++++-- drivers/cpuidle/cpuidle-psci.c | 26 +++++------- drivers/cpuidle/cpuidle-qcom-spm.c | 4 +- drivers/cpuidle/cpuidle-riscv-sbi.c | 19 +++++---- drivers/cpuidle/cpuidle-tegra.c | 31 +++++++++----- drivers/cpuidle/cpuidle.c | 70 ++++++++++++++++++++++--------- drivers/cpuidle/dt_idle_states.c | 2 +- drivers/cpuidle/poll_state.c | 10 ++++- drivers/idle/intel_idle.c | 19 ++++----- drivers/perf/arm_pmu.c | 11 +---- drivers/perf/riscv_pmu_sbi.c | 8 +--- include/asm-generic/vmlinux.lds.h | 9 ++-- include/linux/clockchips.h | 4 +- include/linux/compiler_types.h | 18 +++++++- include/linux/cpu.h | 3 -- include/linux/cpuidle.h | 32 ++++++++++++++ include/linux/cpumask.h | 4 +- include/linux/percpu-defs.h | 2 +- include/linux/sched/idle.h | 40 +++++++++++++----- include/linux/thread_info.h | 18 +++++++- include/linux/tracepoint.h | 15 ++++++- kernel/context_tracking.c | 12 +++--- kernel/cpu_pm.c | 9 ---- kernel/printk/printk.c | 2 +- kernel/sched/idle.c | 47 ++++++--------------- kernel/time/tick-broadcast-hrtimer.c | 29 ++++++------- kernel/time/tick-broadcast.c | 6 ++- kernel/trace/trace.c | 3 ++ kernel/trace/trace_preemptirq.c | 50 ++++++---------------- lib/ubsan.c | 5 ++- mm/kasan/kasan.h | 4 ++ mm/kasan/shadow.c | 38 +++++++++++++++++ tools/objtool/check.c | 17 ++++++++ 131 files changed, 617 insertions(+), 523 deletions(-)
Peter Zijlstra
maintainerPATCH v3 1/51 x86/perf/amd: Remove tracing from perf_lopwr_cb()
The perf_lopwr_cb() is called from the idle routines; there is no RCU there, we must not enter tracing.
--- arch/x86/events/amd/brs.c | 13 +++++-------- arch/x86/include/asm/perf_event.h | 2 +- 2 files changed, 6 insertions(+), 9 deletions(-)
arch/x86/events/amd/brs.c +5 -8
@@ -41,18 +41,15 @@ static inline unsigned int brs_to(int idreturn MSR_AMD_SAMP_BR_FROM +2 * idx +1 ; } -static inline void set_debug_extn_cfg (u64 val) +static __always_inline void set_debug_extn_cfg(u64 val) {/* bits[4:3] must always be set to 11b */ -wrmsrl (MSR_AMD_DBG_EXTN_CFG , val |3ULL <<3 ); +__wrmsr (MSR_AMD_DBG_EXTN_CFG , val |3ULL <<3 , val >>32 ); } -static inline u64 get_debug_extn_cfg (void ) +static __always_inline u64 get_debug_extn_cfg(void) { -u64 val; - -rdmsrl (MSR_AMD_DBG_EXTN_CFG , val); -return val; +return __rdmsr (MSR_AMD_DBG_EXTN_CFG ); }static bool __init amd_brs_detect (void )
@@ -338,7 +335,7 @@ void amd_pmu_brs_sched_task(struct perf_* called from ACPI processor_idle.c or acpi_pad.c * with interrupts disabled */ -void perf_amd_brs_lopwr_cb (bool lopwr_in) +void noinstr perf_amd_brs_lopwr_cb(bool lopwr_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);union amd_debug_extn_cfg cfg;
arch/x86/include/asm/perf_event.h +1 -1
@@ -554,7 +554,7 @@ extern void perf_amd_brs_lopwr_cb(bool lDECLARE_STATIC_CALL (perf_lopwr_cb, perf_amd_brs_lopwr_cb); -static inline void perf_lopwr_cb (bool lopwr_in) +static __always_inline void perf_lopwr_cb(bool lopwr_in) {static_call_mod (perf_lopwr_cb)(lopwr_in); }
[tip: sched/core] x86/perf/amd: Remove tracing from perf_lopwr_cb()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 1f7c232ee080f01ded1236685a357f8926e8c7d5 Gitweb: https://git.kernel.org/tip/1f7c232ee080f01ded1236685a357f8926e8c7d5 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:15 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:03:21 +01:00
x86/perf/amd: Remove tracing from perf_lopwr_cb()
The perf_lopwr_cb() function is called from the idle routines; there is no RCU there, we must not enter tracing.
Use __always_inline, noidle annotations and existing no-trace methods.
--- arch/x86/events/amd/brs.c | 13 +++++-------- arch/x86/include/asm/perf_event.h | 2 +- 2 files changed, 6 insertions(+), 9 deletions(-)
arch/x86/events/amd/brs.c +5 -8
@@ -41,18 +41,15 @@ static inline unsigned int brs_to(int idx)return MSR_AMD_SAMP_BR_FROM +2 * idx +1 ; } -static inline void set_debug_extn_cfg (u64 val) +static __always_inline void set_debug_extn_cfg(u64 val) {/* bits[4:3] must always be set to 11b */ -wrmsrl (MSR_AMD_DBG_EXTN_CFG , val |3ULL <<3 ); +__wrmsr (MSR_AMD_DBG_EXTN_CFG , val |3ULL <<3 , val >>32 ); } -static inline u64 get_debug_extn_cfg (void ) +static __always_inline u64 get_debug_extn_cfg(void) { -u64 val; - -rdmsrl (MSR_AMD_DBG_EXTN_CFG , val); -return val; +return __rdmsr (MSR_AMD_DBG_EXTN_CFG ); }static bool __init amd_brs_detect (void )
@@ -405,7 +402,7 @@ void amd_pmu_brs_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_i* called from ACPI processor_idle.c or acpi_pad.c * with interrupts disabled */ -void perf_amd_brs_lopwr_cb (bool lopwr_in) +void noinstr perf_amd_brs_lopwr_cb(bool lopwr_in) {struct cpu_hw_events *cpuc =this_cpu_ptr (&cpu_hw_events);union amd_debug_extn_cfg cfg;
arch/x86/include/asm/perf_event.h +1 -1
@@ -578,7 +578,7 @@ extern void perf_amd_brs_lopwr_cb(bool lopwr_in);DECLARE_STATIC_CALL (perf_lopwr_cb, perf_amd_brs_lopwr_cb); -static inline void perf_lopwr_cb (bool lopwr_in) +static __always_inline void perf_lopwr_cb(bool lopwr_in) {static_call_mod (perf_lopwr_cb)(lopwr_in); }
Peter Zijlstra
maintainerPATCH v3 2/51 x86/idle: Replace x86_idle with a static_call
In reply to Peter ZijlstraTypical boot time setup; no need to suffer an indirect call for that.
--- arch/x86/kernel/process.c | 50 +++++++++++++++++++++++++--------------------- 1 file changed, 28 insertions(+), 22 deletions(-)
arch/x86/kernel/process.c +28 -22
@@ -24,6 +24,7 @@#include <linux/cpuidle.h> #include <linux/acpi.h> #include <linux/elf-randomize.h> +#include <linux/static_call.h> #include <trace/events/power.h> #include <linux/hw_breakpoint.h> #include <asm/cpu.h>
@@ -692,7 +693,23 @@ void __switch_to_xtra(struct task_structunsigned long boot_option_idle_override =IDLE_NO_OVERRIDE ;EXPORT_SYMBOL (boot_option_idle_override); -static void (*x86_idle)(void ); +/* +* We use this if we don't have any better idle routine.. +*/ +void __cpuidle default_idle(void) +{ +raw_safe_halt (); +} +#if defined(CONFIG_APM_MODULE ) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE ) +EXPORT_SYMBOL (default_idle); +#endif + +DEFINE_STATIC_CALL_NULL (x86_idle, default_idle); + +static bool x86_idle_set (void ) +{ +return !!static_call_query (x86_idle); +}#ifndef CONFIG_SMP static inline void play_dead (void )
@@ -715,28 +732,17 @@ void arch_cpu_idle_dead(void)/* * Called from the generic idle code. */ -void arch_cpu_idle (void ) -{ -x86_idle (); -} - -/* -* We use this if we don't have any better idle routine.. -*/ -void __cpuidle default_idle(void) +void __cpuidle arch_cpu_idle(void) { -raw_safe_halt (); +static_call (x86_idle)(); } -#if defined(CONFIG_APM_MODULE ) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE ) -EXPORT_SYMBOL (default_idle); -#endif #ifdef CONFIG_XEN bool xen_set_default_idle (void ) { -bool ret = !!x86_idle; +bool ret =x86_idle_set (); - x86_idle = default_idle; +static_call_update (x86_idle, default_idle);return ret; }
@@ -859,20 +865,20 @@ void select_idle_routine(const struct cpif (boot_option_idle_override ==IDLE_POLL && smp_num_siblings >1 )pr_warn_once ("WARNING: polling idle and HT enabled, performance may degrade\n" );#endif -if (x86_idle || boot_option_idle_override ==IDLE_POLL ) +if (x86_idle_set () || boot_option_idle_override ==IDLE_POLL )return ;if (boot_cpu_has_bug (X86_BUG_AMD_E400 )) {pr_info ("using AMD E400 aware idle routine\n" ); - x86_idle = amd_e400_idle; +static_call_update (x86_idle, amd_e400_idle); }else if (prefer_mwait_c1_over_halt (c)) {pr_info ("using mwait in idle threads\n" ); - x86_idle = mwait_idle; +static_call_update (x86_idle, mwait_idle); }else if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) {pr_info ("using TDX aware idle routine\n" ); - x86_idle = tdx_safe_halt; +static_call_update (x86_idle, tdx_safe_halt); }else - x86_idle = default_idle; +static_call_update (x86_idle, default_idle); }void amd_e400_c1e_apic_setup (void )
@@ -925,7 +931,7 @@ static int __init idle_setup(char *str)* To continue to load the CPU idle driver, don't touch * the boot_option_idle_override. */ - x86_idle = default_idle; +static_call_update (x86_idle, default_idle); boot_option_idle_override =IDLE_HALT ; }else if (!strcmp (str,"nomwait" )) {/*
[tip: sched/core] x86/idle: Replace 'x86_idle' function pointer with a static_call
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: aaa3896b969bdf946e0fd6aa6199da17b5da8d5c Gitweb: https://git.kernel.org/tip/aaa3896b969bdf946e0fd6aa6199da17b5da8d5c Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:16 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:03:21 +01:00
x86/idle: Replace 'x86_idle' function pointer with a static_call
Typical boot time setup; no need to suffer an indirect call for that.
--- arch/x86/kernel/process.c | 50 +++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 22 deletions(-)
arch/x86/kernel/process.c +28 -22
@@ -24,6 +24,7 @@#include <linux/cpuidle.h> #include <linux/acpi.h> #include <linux/elf-randomize.h> +#include <linux/static_call.h> #include <trace/events/power.h> #include <linux/hw_breakpoint.h> #include <asm/cpu.h>
@@ -694,7 +695,23 @@ void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p)unsigned long boot_option_idle_override =IDLE_NO_OVERRIDE ;EXPORT_SYMBOL (boot_option_idle_override); -static void (*x86_idle)(void ); +/* +* We use this if we don't have any better idle routine.. +*/ +void __cpuidle default_idle(void) +{ +raw_safe_halt (); +} +#if defined(CONFIG_APM_MODULE ) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE ) +EXPORT_SYMBOL (default_idle); +#endif + +DEFINE_STATIC_CALL_NULL (x86_idle, default_idle); + +static bool x86_idle_set (void ) +{ +return !!static_call_query (x86_idle); +}#ifndef CONFIG_SMP static inline void play_dead (void )
@@ -717,28 +734,17 @@ void arch_cpu_idle_dead(void)/* * Called from the generic idle code. */ -void arch_cpu_idle (void ) -{ -x86_idle (); -} - -/* -* We use this if we don't have any better idle routine.. -*/ -void __cpuidle default_idle(void) +void __cpuidle arch_cpu_idle(void) { -raw_safe_halt (); +static_call (x86_idle)(); } -#if defined(CONFIG_APM_MODULE ) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE ) -EXPORT_SYMBOL (default_idle); -#endif #ifdef CONFIG_XEN bool xen_set_default_idle (void ) { -bool ret = !!x86_idle; +bool ret =x86_idle_set (); - x86_idle = default_idle; +static_call_update (x86_idle, default_idle);return ret; }
@@ -880,20 +886,20 @@ void select_idle_routine(const struct cpuinfo_x86 *c)if (boot_option_idle_override ==IDLE_POLL && smp_num_siblings >1 )pr_warn_once ("WARNING: polling idle and HT enabled, performance may degrade\n" );#endif -if (x86_idle || boot_option_idle_override ==IDLE_POLL ) +if (x86_idle_set () || boot_option_idle_override ==IDLE_POLL )return ;if (boot_cpu_has_bug (X86_BUG_AMD_E400 )) {pr_info ("using AMD E400 aware idle routine\n" ); - x86_idle = amd_e400_idle; +static_call_update (x86_idle, amd_e400_idle); }else if (prefer_mwait_c1_over_halt (c)) {pr_info ("using mwait in idle threads\n" ); - x86_idle = mwait_idle; +static_call_update (x86_idle, mwait_idle); }else if (cpu_feature_enabled (X86_FEATURE_TDX_GUEST )) {pr_info ("using TDX aware idle routine\n" ); - x86_idle = tdx_safe_halt; +static_call_update (x86_idle, tdx_safe_halt); }else - x86_idle = default_idle; +static_call_update (x86_idle, default_idle); }void amd_e400_c1e_apic_setup (void )
@@ -946,7 +952,7 @@ static int __init idle_setup(char *str)* To continue to load the CPU idle driver, don't touch * the boot_option_idle_override. */ - x86_idle = default_idle; +static_call_update (x86_idle, default_idle); boot_option_idle_override =IDLE_HALT ; }else if (!strcmp (str,"nomwait" )) {/*
Peter Zijlstra
maintainerPATCH v3 3/51 cpuidle/poll: Ensure IRQ state is invariant
In reply to Peter Zijlstracpuidle_state::enter() methods should be IRQ invariant.
Additionally make sure to use raw_local_irq_*() methods since this cpuidle callback will be called with RCU already disabled.
--- drivers/cpuidle/poll_state.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/cpuidle/poll_state.c +3 -1
@@ -17,7 +17,7 @@ static int __cpuidle poll_idle(struct cp dev->poll_time_limit = false; -local_irq_enable (); +raw_local_irq_enable ();if (!current_set_polling_and_test ()) {unsigned int loop_count =0 ;u64 limit;
@@ -36,6 +36,8 @@ static int __cpuidle poll_idle(struct cp } } } +raw_local_irq_disable (); +current_clr_polling ();return index;
[tip: sched/core] cpuidle/poll: Ensure IRQs stay disabled after cpuidle_state::enter() calls
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5e26aa93391195a64871db5d96d7163f0062ca4f Gitweb: https://git.kernel.org/tip/5e26aa93391195a64871db5d96d7163f0062ca4f Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:17 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:03:21 +01:00
cpuidle/poll: Ensure IRQs stay disabled after cpuidle_state::enter() calls
Make cpuidle_state::enter() methods IRQ state invariant on exit.
Additionally make sure to use raw_local_irq_*() methods since this cpuidle callback will be called with RCU already disabled.
--- drivers/cpuidle/poll_state.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/cpuidle/poll_state.c +3 -1
@@ -17,7 +17,7 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, dev->poll_time_limit = false; -local_irq_enable (); +raw_local_irq_enable ();if (!current_set_polling_and_test ()) {unsigned int loop_count =0 ;u64 limit;
@@ -36,6 +36,8 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, } } } +raw_local_irq_disable (); +current_clr_polling ();return index;
Peter Zijlstra
maintainerPATCH v3 4/51 cpuidle: Move IRQ state validation
In reply to Peter ZijlstraMake cpuidle_enter_state() consistent with the s2idle variant and verify ->enter() always returns with interrupts disabled.
--- drivers/cpuidle/cpuidle.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/cpuidle/cpuidle.c +5 -5
@@ -236,7 +236,11 @@ int cpuidle_enter_state(struct cpuidle_dstop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE ))ct_idle_enter (); + entered_state = target_state->enter (dev, drv, index); +if (WARN_ONCE (!irqs_disabled (),"%ps leaked IRQ state" , target_state->enter)) +raw_local_irq_disable (); +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE ))ct_idle_exit ();start_critical_timings ();
@@ -248,12 +252,8 @@ int cpuidle_enter_state(struct cpuidle_d/* The cpu is no longer idle or about to enter idle. */ sched_idle_set_state (NULL ); -if (broadcast) { -if (WARN_ON_ONCE (!irqs_disabled ())) -local_irq_disable (); - +if (broadcast)tick_broadcast_exit (); - }if (!cpuidle_state_is_coupled (drv, index))local_irq_enable ();
[tip: sched/core] cpuidle: Move IRQ state validation
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: bb7b11258561e47abbacebf76e3ce4092953dfdf Gitweb: https://git.kernel.org/tip/bb7b11258561e47abbacebf76e3ce4092953dfdf Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:18 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:03:22 +01:00
cpuidle: Move IRQ state validation
Make cpuidle_enter_state() consistent with the s2idle variant and verify ->enter() always returns with interrupts disabled.
--- drivers/cpuidle/cpuidle.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
drivers/cpuidle/cpuidle.c +5 -5
@@ -236,7 +236,11 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,stop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE ))ct_idle_enter (); + entered_state = target_state->enter (dev, drv, index); +if (WARN_ONCE (!irqs_disabled (),"%ps leaked IRQ state" , target_state->enter)) +raw_local_irq_disable (); +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE ))ct_idle_exit ();start_critical_timings ();
@@ -248,12 +252,8 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,/* The cpu is no longer idle or about to enter idle. */ sched_idle_set_state (NULL ); -if (broadcast) { -if (WARN_ON_ONCE (!irqs_disabled ())) -local_irq_disable (); - +if (broadcast)tick_broadcast_exit (); - }if (!cpuidle_state_is_coupled (drv, index))local_irq_enable ();
Peter Zijlstra
maintainerPATCH v3 5/51 cpuidle,riscv: Push RCU-idle into driver
In reply to Peter ZijlstraDoing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is daft.
That is, once implicitly through the cpu_pm_*() calls and once explicitly doing ct_irq_*_irqon().
--- drivers/cpuidle/cpuidle-riscv-sbi.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/cpuidle/cpuidle-riscv-sbi.c +5 -4
@@ -116,12 +116,12 @@ static int __sbi_enter_domain_idle_statereturn -1 ;/* Do runtime PM to manage a hierarchical CPU toplogy. */ -ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_suspend (pd_dev);else pm_runtime_put_sync_suspend (pd_dev); -ct_irq_exit_irqson (); + +ct_idle_enter ();if (sbi_is_domain_state_available ()) state =sbi_get_domain_state ();
@@ -130,12 +130,12 @@ static int __sbi_enter_domain_idle_state ret =sbi_suspend (state) ?-1 : idx; -ct_irq_enter_irqson (); +ct_idle_exit (); +if (s2idle)dev_pm_genpd_resume (pd_dev);else pm_runtime_get_sync (pd_dev); -ct_irq_exit_irqson ();cpu_pm_exit ();
@@ -246,6 +246,7 @@ static int sbi_dt_cpu_init_topology(stru* of a shared state for the domain, assumes the domain states are all * deeper states. */ + drv->states[state_count -1 ].flags |=CPUIDLE_FLAG_RCU_IDLE ; drv->states[state_count -1 ].enter = sbi_enter_domain_idle_state; drv->states[state_count -1 ].enter_s2idle = sbi_enter_s2idle_domain_idle_state;
[tip: sched/core] cpuidle, riscv: Push RCU-idle into driver
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 8e9ab9e8da1eae61fdff35690d998eaf8cd527dc Gitweb: https://git.kernel.org/tip/8e9ab9e8da1eae61fdff35690d998eaf8cd527dc Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:19 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:03:22 +01:00
cpuidle, riscv: Push RCU-idle into driver
Doing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is suboptimal.
That is, once implicitly through the cpu_pm_*() calls and once explicitly doing ct_irq_*_irqon().
--- drivers/cpuidle/cpuidle-riscv-sbi.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/cpuidle/cpuidle-riscv-sbi.c +5 -4
@@ -121,12 +121,12 @@ static int __sbi_enter_domain_idle_state(struct cpuidle_device *dev,return -1 ;/* Do runtime PM to manage a hierarchical CPU toplogy. */ -ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_suspend (pd_dev);else pm_runtime_put_sync_suspend (pd_dev); -ct_irq_exit_irqson (); + +ct_idle_enter ();if (sbi_is_domain_state_available ()) state =sbi_get_domain_state ();
@@ -135,12 +135,12 @@ static int __sbi_enter_domain_idle_state(struct cpuidle_device *dev, ret =sbi_suspend (state) ?-1 : idx; -ct_irq_enter_irqson (); +ct_idle_exit (); +if (s2idle)dev_pm_genpd_resume (pd_dev);else pm_runtime_get_sync (pd_dev); -ct_irq_exit_irqson ();cpu_pm_exit ();
@@ -251,6 +251,7 @@ static int sbi_dt_cpu_init_topology(struct cpuidle_driver *drv,* of a shared state for the domain, assumes the domain states are all * deeper states. */ + drv->states[state_count -1 ].flags |=CPUIDLE_FLAG_RCU_IDLE ; drv->states[state_count -1 ].enter = sbi_enter_domain_idle_state; drv->states[state_count -1 ].enter_s2idle = sbi_enter_s2idle_domain_idle_state;
Peter Zijlstra
maintainerPATCH v3 6/51 cpuidle,tegra: Push RCU-idle into driver
In reply to Peter ZijlstraDoing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is daft.
Notably once implicitly through the cpu_pm_*() calls and once explicitly doing RCU_NONIDLE().
--- drivers/cpuidle/cpuidle-tegra.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
drivers/cpuidle/cpuidle-tegra.c +16 -5
@@ -180,9 +180,11 @@ static int tegra_cpuidle_state_enter(str }local_fiq_disable (); -RCU_NONIDLE (tegra_pm_set_cpu_in_lp2 ()); +tegra_pm_set_cpu_in_lp2 ();cpu_pm_enter (); +ct_idle_enter (); +switch (index) {case TEGRA_C7 : err =tegra_cpuidle_c7_enter ();
@@ -197,8 +199,10 @@ static int tegra_cpuidle_state_enter(strbreak ; } +ct_idle_exit (); +cpu_pm_exit (); -RCU_NONIDLE (tegra_pm_clear_cpu_in_lp2 ()); +tegra_pm_clear_cpu_in_lp2 ();local_fiq_enable ();return err ?: index;
@@ -226,6 +230,7 @@ static int tegra_cpuidle_enter(struct cpstruct cpuidle_driver *drv, int index) { + bool do_rcu = drv->states[index].flags & CPUIDLE_FLAG_RCU_IDLE;unsigned int cpu =cpu_logical_map (dev->cpu);int ret;
@@ -233,9 +238,13 @@ static int tegra_cpuidle_enter(struct cpif (dev->states_usage[index].disable)return -1 ; -if (index ==TEGRA_C1 ) +if (index ==TEGRA_C1 ) { +if (do_rcu) +ct_idle_enter (); ret =arm_cpuidle_simple_enter (dev, drv, index); -else +if (do_rcu) +ct_idle_exit (); + }else ret =tegra_cpuidle_state_enter (dev, index, cpu);if (ret <0 ) {
@@ -285,7 +294,8 @@ static struct cpuidle_driver tegra_idle_ .exit_latency = 2000, .target_residency = 2200, .power_usage = 100, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .name = "C7", .desc = "CPU core powered off", },
@@ -295,6 +305,7 @@ static struct cpuidle_driver tegra_idle_ .target_residency = 10000, .power_usage = 0, .flags =CPUIDLE_FLAG_TIMER_STOP | +CPUIDLE_FLAG_RCU_IDLE | CPUIDLE_FLAG_COUPLED, .name = "CC6", .desc = "CPU cluster powered off",
[tip: sched/core] cpuidle, tegra: Push RCU-idle into driver
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5fca0d9f5d76664786ca6c09076341def165a677 Gitweb: https://git.kernel.org/tip/5fca0d9f5d76664786ca6c09076341def165a677 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:20 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:03:22 +01:00
cpuidle, tegra: Push RCU-idle into driver
Doing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is suboptimal.
Notably once implicitly through the cpu_pm_*() calls and once explicitly doing RCU_NONIDLE().
--- drivers/cpuidle/cpuidle-tegra.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
drivers/cpuidle/cpuidle-tegra.c +16 -5
@@ -180,9 +180,11 @@ static int tegra_cpuidle_state_enter(struct cpuidle_device *dev, }local_fiq_disable (); -RCU_NONIDLE (tegra_pm_set_cpu_in_lp2 ()); +tegra_pm_set_cpu_in_lp2 ();cpu_pm_enter (); +ct_idle_enter (); +switch (index) {case TEGRA_C7 : err =tegra_cpuidle_c7_enter ();
@@ -197,8 +199,10 @@ static int tegra_cpuidle_state_enter(struct cpuidle_device *dev,break ; } +ct_idle_exit (); +cpu_pm_exit (); -RCU_NONIDLE (tegra_pm_clear_cpu_in_lp2 ()); +tegra_pm_clear_cpu_in_lp2 ();local_fiq_enable ();return err ?: index;
@@ -226,6 +230,7 @@ static int tegra_cpuidle_enter(struct cpuidle_device *dev,struct cpuidle_driver *drv, int index) { + bool do_rcu = drv->states[index].flags & CPUIDLE_FLAG_RCU_IDLE;unsigned int cpu =cpu_logical_map (dev->cpu);int ret;
@@ -233,9 +238,13 @@ static int tegra_cpuidle_enter(struct cpuidle_device *dev,if (dev->states_usage[index].disable)return -1 ; -if (index ==TEGRA_C1 ) +if (index ==TEGRA_C1 ) { +if (do_rcu) +ct_idle_enter (); ret =arm_cpuidle_simple_enter (dev, drv, index); -else +if (do_rcu) +ct_idle_exit (); + }else ret =tegra_cpuidle_state_enter (dev, index, cpu);if (ret <0 ) {
@@ -285,7 +294,8 @@ static struct cpuidle_driver tegra_idle_driver = { .exit_latency = 2000, .target_residency = 2200, .power_usage = 100, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .name = "C7", .desc = "CPU core powered off", },
@@ -295,6 +305,7 @@ static struct cpuidle_driver tegra_idle_driver = { .target_residency = 10000, .power_usage = 0, .flags =CPUIDLE_FLAG_TIMER_STOP | +CPUIDLE_FLAG_RCU_IDLE | CPUIDLE_FLAG_COUPLED, .name = "CC6", .desc = "CPU cluster powered off",
Peter Zijlstra
maintainerPATCH v3 7/51 cpuidle,psci: Push RCU-idle into driver
In reply to Peter ZijlstraDoing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is daft.
Notably once implicitly through the cpu_pm_*() calls and once explicitly doing ct_irq_*_irqon().
--- drivers/cpuidle/cpuidle-psci.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/cpuidle/cpuidle-psci.c +5 -4
@@ -69,12 +69,12 @@ static int __psci_enter_domain_idle_statreturn -1 ;/* Do runtime PM to manage a hierarchical CPU toplogy. */ -ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_suspend (pd_dev);else pm_runtime_put_sync_suspend (pd_dev); -ct_irq_exit_irqson (); + +ct_idle_enter (); state =psci_get_domain_state ();if (!state)
@@ -82,12 +82,12 @@ static int __psci_enter_domain_idle_stat ret =psci_cpu_suspend_enter (state) ?-1 : idx; -ct_irq_enter_irqson (); +ct_idle_exit (); +if (s2idle)dev_pm_genpd_resume (pd_dev);else pm_runtime_get_sync (pd_dev); -ct_irq_exit_irqson ();cpu_pm_exit ();
@@ -240,6 +240,7 @@ static int psci_dt_cpu_init_topology(str* of a shared state for the domain, assumes the domain states are all * deeper states. */ + drv->states[state_count -1 ].flags |=CPUIDLE_FLAG_RCU_IDLE ; drv->states[state_count -1 ].enter = psci_enter_domain_idle_state; drv->states[state_count -1 ].enter_s2idle = psci_enter_s2idle_domain_idle_state; psci_cpuidle_use_cpuhp = true;
[tip: sched/core] cpuidle, psci: Push RCU-idle into driver
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e038f7b8028a1d1bc8ac82351c71ea538f19a879 Gitweb: https://git.kernel.org/tip/e038f7b8028a1d1bc8ac82351c71ea538f19a879 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:21 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:03:22 +01:00
cpuidle, psci: Push RCU-idle into driver
Doing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is suboptimal.
Notably once implicitly through the cpu_pm_*() calls and once explicitly doing ct_irq_*_irqon().
--- drivers/cpuidle/cpuidle-psci.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/cpuidle/cpuidle-psci.c +5 -4
@@ -69,12 +69,12 @@ static int __psci_enter_domain_idle_state(struct cpuidle_device *dev,return -1 ;/* Do runtime PM to manage a hierarchical CPU toplogy. */ -ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_suspend (pd_dev);else pm_runtime_put_sync_suspend (pd_dev); -ct_irq_exit_irqson (); + +ct_idle_enter (); state =psci_get_domain_state ();if (!state)
@@ -82,12 +82,12 @@ static int __psci_enter_domain_idle_state(struct cpuidle_device *dev, ret =psci_cpu_suspend_enter (state) ?-1 : idx; -ct_irq_enter_irqson (); +ct_idle_exit (); +if (s2idle)dev_pm_genpd_resume (pd_dev);else pm_runtime_get_sync (pd_dev); -ct_irq_exit_irqson ();cpu_pm_exit ();
@@ -240,6 +240,7 @@ static int psci_dt_cpu_init_topology(struct cpuidle_driver *drv,* of a shared state for the domain, assumes the domain states are all * deeper states. */ + drv->states[state_count -1 ].flags |=CPUIDLE_FLAG_RCU_IDLE ; drv->states[state_count -1 ].enter = psci_enter_domain_idle_state; drv->states[state_count -1 ].enter_s2idle = psci_enter_s2idle_domain_idle_state; psci_cpuidle_use_cpuhp = true;
Geert Uytterhoeven
maintainerRe: [PATCH v3 07/51] cpuidle,psci: Push RCU-idle into driver
In reply to Peter ZijlstraHoi Peter,
(reduced the insane CC list)
Quoted 6 lines · Peter Zijlstra
Doing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is daft.
Notably once implicitly through the cpu_pm_*() calls and once explicitly doing ct_irq_*_irqon().
Quoted 7 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Frederic Weisbecker <[email protected]>Reviewed-by Guo Ren <[email protected]>Tested-by Kajetan Puchalski <[email protected]>Tested-by Tony Lindgren <[email protected]>Tested-by Ulf Hansson <[email protected]>
Thanks for your patch, which is now commit e038f7b8028a1d1b ("cpuidle,
psci: Push RCU-idle into driver") in v6.3-rc1.
I have bisected a PSCI checker regression on Renesas R-Car Gen3/4 SoCs to commit a01353cf1896ea5b ("cpuidle: Fix ct_idle_*() usage") (the 7 commits before that do not compile):
psci_checker: PSCI checker started using 2 CPUspsci_checker: Starting hotplug testspsci_checker: Trying to turn off and on again all CPUs
psci: CPU0 killed (polled 0 ms)
Detected PIPT I-cache on CPU0
CPU0: Booted secondary processor 0x0000000000 [0x411fd073]psci_checker: Trying to turn off and on again group 0 (CPUs 0-1)
psci: CPU0 killed (polled 0 ms)
Detected PIPT I-cache on CPU0
CPU0: Booted secondary processor 0x0000000000 [0x411fd073]psci_checker: Hotplug tests passed OKpsci_checker: Starting suspend tests (10 cycles per state)psci_checker: CPU 0 entering suspend cycles, states 1 through 1psci_checker: CPU 1 entering suspend cycles, states 1 through 1
------------[ cut here ]------------ WARNING: CPU: 1 PID: 177 at kernel/context_tracking.c:141 ct_kernel_exit.constprop.0+0xd8/0xf4 Modules linked in: CPU: 1 PID: 177 Comm: psci_suspend_te Not tainted 6.2.0-rc1-salvator-x-00052-ga01353cf1896 #1415 Hardware name: Renesas Salvator-X 2nd version board based on r8a77965 (DT) pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : ct_kernel_exit.constprop.0+0xd8/0xf4 lr : ct_kernel_exit.constprop.0+0xc8/0xf4
sp : ffffffc00b73bd30
x29: ffffffc00b73bd30 x28: ffffff807fbadc90 x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
x23: ffffff800981e140 x22: 0000000000000001 x21: 0000000000010000
x20: ffffffc0086be1d8 x19: ffffff807fbac070 x18: 0000000000000000
x17: ffffff80083d1000 x16: ffffffc00841fff8 x15: ffffffc00b73b990
x14: ffffffc00895be78 x13: 0000000000000001 x12: 0000000000000000
x11: 00000000000001aa x10: 00000000ffffffea x9 : 000000000000000f x8 : ffffffc00b73bb68 x7 : ffffffc00b73be18 x6 : ffffffc00815ff34 x5 : ffffffc00a6a0c30 x4 : ffffffc00801ce00 x3 : 0000000000000000 x2 : ffffffc008dc3070 x1 : ffffffc008dc3078 x0 : 0000000004208040
Call trace: ct_kernel_exit.constprop.0+0xd8/0xf4 ct_idle_enter+0x18/0x20 psci_enter_idle_state+0xa4/0xfc suspend_test_thread+0x238/0x2f0 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20 irq event stamp: 0 hardirqs last enabled at (0): [<0000000000000000>] 0x0 hardirqs last disabled at (0): [<ffffffc0080798b0>] copy_process+0x608/0x13dc softirqs last enabled at (0): [<ffffffc0080798b0>] copy_process+0x608/0x13dc softirqs last disabled at (0): [<0000000000000000>] 0x0 ---[ end trace 0000000000000000 ]--- ------------[ cut here ]------------ WARNING: CPU: 1 PID: 177 at kernel/context_tracking.c:186 ct_kernel_enter.constprop.0+0x78/0xa4 Modules linked in: CPU: 1 PID: 177 Comm: psci_suspend_te Tainted: G W 6.2.0-rc1-salvator-x-00052-ga01353cf1896 #1415 Hardware name: Renesas Salvator-X 2nd version board based on r8a77965 (DT) pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : ct_kernel_enter.constprop.0+0x78/0xa4 lr : ct_kernel_enter.constprop.0+0x68/0xa4
sp : ffffffc00b73bd30
x29: ffffffc00b73bd30 x28: ffffff807fbadc90 x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
x23: ffffff800981e140 x22: 0000000000000001 x21: 00000000ffffffa1
x20: ffffffc0086be1d8 x19: 00000000000000c0 x18: 0000000000000000
x17: ffffff80083d1000 x16: ffffffc00841fff8 x15: ffffffc00b73b990
x14: ffffffc00895be78 x13: ffffff800e325180 x12: ffffffc076de9000
x11: 0000000034d4d91d x10: 0000000000000008 x9 : 0000000000001000 x8 : ffffffc008012800 x7 : 0000000000000000 x6 : ffffff807fbac070 x5 : ffffffc008dc3070 x4 : 0000000000000000 x3 : 000000000001a9fc x2 : 0000000000000003 x1 : ffffffc008dc3070 x0 : 0000000004208040
Call trace: ct_kernel_enter.constprop.0+0x78/0xa4 ct_idle_exit+0x18/0x38 psci_enter_idle_state+0xdc/0xfc suspend_test_thread+0x238/0x2f0 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20 irq event stamp: 0 hardirqs last enabled at (0): [<0000000000000000>] 0x0 hardirqs last disabled at (0): [<ffffffc0080798b0>] copy_process+0x608/0x13dc softirqs last enabled at (0): [<ffffffc0080798b0>] copy_process+0x608/0x13dc softirqs last disabled at (0): [<0000000000000000>] 0x0 ---[ end trace 0000000000000000 ]---
psci_checker: Failed to suspend CPU 1: error -1 (requested state 1, cycle 0)psci_checker: CPU 0 suspend test results: success 0, shallow states 10, errors 0
mmcblk0rpmb: mmc0:0001 BGSD3R 4.00 MiB, chardev (243:0)psci_checker: CPU 1 suspend test results: success 0, shallow states 9, errors 1psci_checker: 1 error(s) encountered in suspend testspsci_checker: PSCI checker completed
Quoted 37 lines
---
drivers/cpuidle/cpuidle-psci.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)drivers/cpuidle/cpuidle-psci.c +4
-4@@ -69,12 +69,12 @@ static int __psci_enter_domain_idle_statreturn -1 ;/* Do runtime PM to manage a hierarchical CPU toplogy. */ -ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_suspend (pd_dev);else pm_runtime_put_sync_suspend (pd_dev); -ct_irq_exit_irqson (); + +ct_idle_enter (); state =psci_get_domain_state ();if (!state)@@ -82,12 +82,12 @@ static int __psci_enter_domain_idle_stat ret =psci_cpu_suspend_enter (state) ?-1 : idx; -ct_irq_enter_irqson (); +ct_idle_exit (); +if (s2idle)dev_pm_genpd_resume (pd_dev);else pm_runtime_get_sync (pd_dev); -ct_irq_exit_irqson ();cpu_pm_exit ();
drivers/cpuidle/cpuidle-psci.c static int psci_dt_cpu_init_topology(str · line 246
Quoted 7 lines
drivers/cpuidle/cpuidle-psci.c +1
-0@@ -240,6 +240,7 @@ static int psci_dt_cpu_init_topology(str* of a shared state for the domain, assumes the domain states are all * deeper states. */ + drv->states[state_count -1 ].flags |=CPUIDLE_FLAG_RCU_IDLE ; drv->states[state_count -1 ].enter = psci_enter_domain_idle_state; drv->states[state_count -1 ].enter_s2idle = psci_enter_s2idle_domain_idle_state; psci_cpuidle_use_cpuhp = true;
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Mark Rutland
maintainerRe: [PATCH v3 07/51] cpuidle,psci: Push RCU-idle into driver
Hi Geert,
Quoted 3 lines · Geert Uytterhoeven
Hoi Peter,
(reduced the insane CC list)
Helpfully you dropped me from Cc, so I missed this until just now...
Quoted 36 lines · Peter Zijlstra
Quoted 6 lines
Doing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is daft.
Notably once implicitly through the cpu_pm_*() calls and once explicitly doing ct_irq_*_irqon().
Quoted 7 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Frederic Weisbecker <[email protected]>Reviewed-by Guo Ren <[email protected]>Tested-by Kajetan Puchalski <[email protected]>Tested-by Tony Lindgren <[email protected]>Tested-by Ulf Hansson <[email protected]>Thanks for your patch, which is now commit
e038f7b8028a1d1b("cpuidle,
psci: Push RCU-idle into driver") in v6.3-rc1.I have bisected a PSCI checker regression on Renesas R-Car Gen3/4 SoCs to commit
a01353cf1896ea5b("cpuidle: Fix ct_idle_*() usage") (the 7 commits before that do not compile):
psci_checker: PSCI checker started using 2 CPUspsci_checker: Starting hotplug testspsci_checker: Trying to turn off and on again all CPUs
psci: CPU0 killed (polled 0 ms)
Detected PIPT I-cache on CPU0
CPU0: Booted secondary processor0x0000000000[0x411fd073]psci_checker: Trying to turn off and on again group 0 (CPUs 0-1)
psci: CPU0 killed (polled 0 ms)
Detected PIPT I-cache on CPU0
CPU0: Booted secondary processor0x0000000000[0x411fd073]psci_checker: Hotplug tests passed OKpsci_checker: Starting suspend tests (10 cycles per state)psci_checker: CPU 0 entering suspend cycles, states 1 through 1psci_checker: CPU 1 entering suspend cycles, states 1 through 1
Quoted 2 lines
------------[ cut here ]------------ WARNING: CPU: 1 PID: 177 at kernel/context_tracking.c:141 ct_kernel_exit.constprop.0+0xd8/0xf4
So that's:
WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
... and the PSCI checker doens't run in the context of the idle thread, so the warning is correct, and we're violating the expectation of the context tracking code.
The PSCI checker is very much a special case, and I'm not sure how we can fix this without removing the warning in the cases we want it. It'd be nicer if we could "queue" the idle into the relevant idle thread. :/
I'm very tempted to say we should just rip the checker code out, rather than contorting the rest of the code to make that work.
Thanks,
Mark.
Quoted 125 lines
Modules linked in: CPU: 1 PID: 177 Comm: psci_suspend_te Not tainted 6.2.0-rc1-salvator-x-00052-ga01353cf1896 #1415 Hardware name: Renesas Salvator-X 2nd version board based on r8a77965 (DT) pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : ct_kernel_exit.constprop.0+0xd8/0xf4 lr : ct_kernel_exit.constprop.0+0xc8/0xf4sp :
ffffffc00b73bd30
x29:ffffffc00b73bd30x28:ffffff807fbadc90x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
x23:ffffff800981e140x22: 0000000000000001 x21: 0000000000010000
x20:ffffffc0086be1d8x19:ffffff807fbac070x18: 0000000000000000
x17:ffffff80083d1000x16:ffffffc00841fff8x15:ffffffc00b73b990
x14:ffffffc00895be78x13: 0000000000000001 x12: 0000000000000000
x11:00000000000001aax10:00000000ffffffeax9 :000000000000000fx8 :ffffffc00b73bb68x7 :ffffffc00b73be18x6 :ffffffc00815ff34x5 :ffffffc00a6a0c30x4 :ffffffc00801ce00x3 : 0000000000000000 x2 :ffffffc008dc3070x1 :ffffffc008dc3078x0 : 0000000004208040Call trace: ct_kernel_exit.constprop.0+0xd8/0xf4 ct_idle_enter+0x18/0x20 psci_enter_idle_state+0xa4/0xfc suspend_test_thread+0x238/0x2f0 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20 irq event stamp: 0 hardirqs last enabled at (0): [<0000000000000000>] 0x0 hardirqs last disabled at (0): [<ffffffc0080798b0>] copy_process+0x608/0x13dc softirqs last enabled at (0): [<ffffffc0080798b0>] copy_process+0x608/0x13dc softirqs last disabled at (0): [<0000000000000000>] 0x0 ---[ end trace 0000000000000000 ]--- ------------[ cut here ]------------ WARNING: CPU: 1 PID: 177 at kernel/context_tracking.c:186 ct_kernel_enter.constprop.0+0x78/0xa4 Modules linked in: CPU: 1 PID: 177 Comm: psci_suspend_te Tainted: G W 6.2.0-rc1-salvator-x-00052-ga01353cf1896 #1415 Hardware name: Renesas Salvator-X 2nd version board based on r8a77965 (DT) pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : ct_kernel_enter.constprop.0+0x78/0xa4 lr : ct_kernel_enter.constprop.0+0x68/0xa4sp :
ffffffc00b73bd30
x29:ffffffc00b73bd30x28:ffffff807fbadc90x27: 0000000000000000
x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
x23:ffffff800981e140x22: 0000000000000001 x21:00000000ffffffa1
x20:ffffffc0086be1d8x19:00000000000000c0x18: 0000000000000000
x17:ffffff80083d1000x16:ffffffc00841fff8x15:ffffffc00b73b990
x14:ffffffc00895be78x13:ffffff800e325180x12:ffffffc076de9000
x11:0000000034d4d91dx10: 0000000000000008 x9 : 0000000000001000 x8 :ffffffc008012800x7 : 0000000000000000 x6 :ffffff807fbac070x5 :ffffffc008dc3070x4 : 0000000000000000 x3 :000000000001a9fcx2 : 0000000000000003 x1 :ffffffc008dc3070x0 : 0000000004208040Call trace: ct_kernel_enter.constprop.0+0x78/0xa4 ct_idle_exit+0x18/0x38 psci_enter_idle_state+0xdc/0xfc suspend_test_thread+0x238/0x2f0 kthread+0xd8/0xe8 ret_from_fork+0x10/0x20 irq event stamp: 0 hardirqs last enabled at (0): [<0000000000000000>] 0x0 hardirqs last disabled at (0): [<ffffffc0080798b0>] copy_process+0x608/0x13dc softirqs last enabled at (0): [<ffffffc0080798b0>] copy_process+0x608/0x13dc softirqs last disabled at (0): [<0000000000000000>] 0x0 ---[ end trace 0000000000000000 ]---
psci_checker: Failed to suspend CPU 1: error -1 (requested state 1, cycle 0)psci_checker: CPU 0 suspend test results: success 0, shallow states 10, errors 0
mmcblk0rpmb: mmc0:0001 BGSD3R 4.00 MiB, chardev (243:0)psci_checker: CPU 1 suspend test results: success 0, shallow states 9, errors 1psci_checker: 1 error(s) encountered in suspend testspsci_checker: PSCI checker completedQuoted 37 lines
---
drivers/cpuidle/cpuidle-psci.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)drivers/cpuidle/cpuidle-psci.c +4
-4@@ -69,12 +69,12 @@ static int __psci_enter_domain_idle_statreturn -1 ;/* Do runtime PM to manage a hierarchical CPU toplogy. */ -ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_suspend (pd_dev);else pm_runtime_put_sync_suspend (pd_dev); -ct_irq_exit_irqson (); + +ct_idle_enter (); state =psci_get_domain_state ();if (!state)@@ -82,12 +82,12 @@ static int __psci_enter_domain_idle_stat ret =psci_cpu_suspend_enter (state) ?-1 : idx; -ct_irq_enter_irqson (); +ct_idle_exit (); +if (s2idle)dev_pm_genpd_resume (pd_dev);else pm_runtime_get_sync (pd_dev); -ct_irq_exit_irqson ();cpu_pm_exit ();drivers/cpuidle/cpuidle-psci.c static int psci_dt_cpu_init_topology(str · line 246
Quoted 7 lines
drivers/cpuidle/cpuidle-psci.c +1
-0@@ -240,6 +240,7 @@ static int psci_dt_cpu_init_topology(str* of a shared state for the domain, assumes the domain states are all * deeper states. */ + drv->states[state_count -1 ].flags |=CPUIDLE_FLAG_RCU_IDLE ; drv->states[state_count -1 ].enter = psci_enter_domain_idle_state; drv->states[state_count -1 ].enter_s2idle = psci_enter_s2idle_domain_idle_state; psci_cpuidle_use_cpuhp = true;Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Peter Zijlstra
maintainerPATCH v3 8/51 cpuidle,imx6: Push RCU-idle into driver
In reply to Peter ZijlstraDoing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is daft.
Notably both cpu_pm_enter() and cpu_cluster_pm_enter() implicity re-enable RCU.
--- arch/arm/mach-imx/cpuidle-imx6sx.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/arm/mach-imx/cpuidle-imx6sx.c +4 -1
@@ -47,7 +47,9 @@ static int imx6sx_enter_wait(struct cpuicpu_pm_enter ();cpu_cluster_pm_enter (); +ct_idle_enter ();cpu_suspend (0 , imx6sx_idle_finish); +ct_idle_exit ();cpu_cluster_pm_exit ();cpu_pm_exit ();
@@ -87,7 +89,8 @@ static struct cpuidle_driver imx6sx_cpui*/ .exit_latency = 300, .target_residency = 500, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .enter = imx6sx_enter_wait, .name = "LOW-POWER-IDLE", .desc = "ARM power off",
[tip: sched/core] cpuidle, ARM/imx6: Push RCU-idle into driver
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b3f46658ce40a3467cda82f920dd9d5325ab0eaf Gitweb: https://git.kernel.org/tip/b3f46658ce40a3467cda82f920dd9d5325ab0eaf Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:22 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:46:40 +01:00
cpuidle, ARM/imx6: Push RCU-idle into driver
Doing RCU-idle outside the driver, only to then temporarily enable it again, at least twice, before going idle is suboptimal.
Notably both cpu_pm_enter() and cpu_cluster_pm_enter() implicity re-enable RCU.
--- arch/arm/mach-imx/cpuidle-imx6sx.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
arch/arm/mach-imx/cpuidle-imx6sx.c +4 -1
@@ -47,7 +47,9 @@ static int imx6sx_enter_wait(struct cpuidle_device *dev,cpu_pm_enter ();cpu_cluster_pm_enter (); +ct_idle_enter ();cpu_suspend (0 , imx6sx_idle_finish); +ct_idle_exit ();cpu_cluster_pm_exit ();cpu_pm_exit ();
@@ -87,7 +89,8 @@ static struct cpuidle_driver imx6sx_cpuidle_driver = {*/ .exit_latency = 300, .target_residency = 500, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .enter = imx6sx_enter_wait, .name = "LOW-POWER-IDLE", .desc = "ARM power off",
Peter Zijlstra
maintainerPATCH v3 9/51 cpuidle,omap3: Push RCU-idle into driver
In reply to Peter ZijlstraDoing RCU-idle outside the driver, only to then teporarily enable it again before going idle is daft.
Notably the cpu_pm_*() calls implicitly re-enable RCU for a bit.
--- arch/arm/mach-omap2/cpuidle34xx.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
arch/arm/mach-omap2/cpuidle34xx.c +16 -0
@@ -133,7 +133,9 @@ static int omap3_enter_idle(struct cpuid }/* Execute ARM wfi */ +ct_idle_enter ();omap_sram_idle (); +ct_idle_exit ();/* * Call idle CPU PM enter notifier chain to restore
@@ -265,6 +267,7 @@ static struct cpuidle_driver omap3_idle_ .owner = THIS_MODULE, .states = { { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 2 + 2, .target_residency = 5,
@@ -272,6 +275,7 @@ static struct cpuidle_driver omap3_idle_ .desc = "MPU ON + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 10 + 10, .target_residency = 30,
@@ -279,6 +283,7 @@ static struct cpuidle_driver omap3_idle_ .desc = "MPU ON + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 50 + 50, .target_residency = 300,
@@ -286,6 +291,7 @@ static struct cpuidle_driver omap3_idle_ .desc = "MPU RET + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 1500 + 1800, .target_residency = 4000,
@@ -293,6 +299,7 @@ static struct cpuidle_driver omap3_idle_ .desc = "MPU OFF + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 2500 + 7500, .target_residency = 12000,
@@ -300,6 +307,7 @@ static struct cpuidle_driver omap3_idle_ .desc = "MPU RET + CORE RET", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 3000 + 8500, .target_residency = 15000,
@@ -307,6 +315,7 @@ static struct cpuidle_driver omap3_idle_ .desc = "MPU OFF + CORE RET", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 10000 + 30000, .target_residency = 30000,
@@ -328,6 +337,7 @@ static struct cpuidle_driver omap3430_id .owner = THIS_MODULE, .states = { { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 110 + 162, .target_residency = 5,
@@ -335,6 +345,7 @@ static struct cpuidle_driver omap3430_id .desc = "MPU ON + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 106 + 180, .target_residency = 309,
@@ -342,6 +353,7 @@ static struct cpuidle_driver omap3430_id .desc = "MPU ON + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 107 + 410, .target_residency = 46057,
@@ -349,6 +361,7 @@ static struct cpuidle_driver omap3430_id .desc = "MPU RET + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 121 + 3374, .target_residency = 46057,
@@ -356,6 +369,7 @@ static struct cpuidle_driver omap3430_id .desc = "MPU OFF + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 855 + 1146, .target_residency = 46057,
@@ -363,6 +377,7 @@ static struct cpuidle_driver omap3430_id .desc = "MPU RET + CORE RET", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 7580 + 4134, .target_residency = 484329,
@@ -370,6 +385,7 @@ static struct cpuidle_driver omap3430_id .desc = "MPU OFF + CORE RET", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 7505 + 15274, .target_residency = 484329,
[tip: sched/core] cpuidle, OMAP3: Push RCU-idle into driver
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4d1be9e745382b41492b0cb9000829863db7133a Gitweb: https://git.kernel.org/tip/4d1be9e745382b41492b0cb9000829863db7133a Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:23 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:47:19 +01:00
cpuidle, OMAP3: Push RCU-idle into driver
Doing RCU-idle outside the driver, only to then teporarily enable it again before going idle is suboptimal.
Notably the cpu_pm_*() calls implicitly re-enable RCU for a bit.
--- arch/arm/mach-omap2/cpuidle34xx.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
arch/arm/mach-omap2/cpuidle34xx.c +16 -0
@@ -133,7 +133,9 @@ static int omap3_enter_idle(struct cpuidle_device *dev, }/* Execute ARM wfi */ +ct_idle_enter ();omap_sram_idle (); +ct_idle_exit ();/* * Call idle CPU PM enter notifier chain to restore
@@ -265,6 +267,7 @@ static struct cpuidle_driver omap3_idle_driver = { .owner = THIS_MODULE, .states = { { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 2 + 2, .target_residency = 5,
@@ -272,6 +275,7 @@ static struct cpuidle_driver omap3_idle_driver = { .desc = "MPU ON + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 10 + 10, .target_residency = 30,
@@ -279,6 +283,7 @@ static struct cpuidle_driver omap3_idle_driver = { .desc = "MPU ON + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 50 + 50, .target_residency = 300,
@@ -286,6 +291,7 @@ static struct cpuidle_driver omap3_idle_driver = { .desc = "MPU RET + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 1500 + 1800, .target_residency = 4000,
@@ -293,6 +299,7 @@ static struct cpuidle_driver omap3_idle_driver = { .desc = "MPU OFF + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 2500 + 7500, .target_residency = 12000,
@@ -300,6 +307,7 @@ static struct cpuidle_driver omap3_idle_driver = { .desc = "MPU RET + CORE RET", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 3000 + 8500, .target_residency = 15000,
@@ -307,6 +315,7 @@ static struct cpuidle_driver omap3_idle_driver = { .desc = "MPU OFF + CORE RET", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 10000 + 30000, .target_residency = 30000,
@@ -328,6 +337,7 @@ static struct cpuidle_driver omap3430_idle_driver = { .owner = THIS_MODULE, .states = { { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 110 + 162, .target_residency = 5,
@@ -335,6 +345,7 @@ static struct cpuidle_driver omap3430_idle_driver = { .desc = "MPU ON + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 106 + 180, .target_residency = 309,
@@ -342,6 +353,7 @@ static struct cpuidle_driver omap3430_idle_driver = { .desc = "MPU ON + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 107 + 410, .target_residency = 46057,
@@ -349,6 +361,7 @@ static struct cpuidle_driver omap3430_idle_driver = { .desc = "MPU RET + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 121 + 3374, .target_residency = 46057,
@@ -356,6 +369,7 @@ static struct cpuidle_driver omap3430_idle_driver = { .desc = "MPU OFF + CORE ON", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 855 + 1146, .target_residency = 46057,
@@ -363,6 +377,7 @@ static struct cpuidle_driver omap3430_idle_driver = { .desc = "MPU RET + CORE RET", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 7580 + 4134, .target_residency = 484329,
@@ -370,6 +385,7 @@ static struct cpuidle_driver omap3430_idle_driver = { .desc = "MPU OFF + CORE RET", }, { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = omap3_enter_idle_bm, .exit_latency = 7505 + 15274, .target_residency = 484329,
Peter Zijlstra
maintainerPATCH v3 10/51 cpuidle,armada: Push RCU-idle into driver
In reply to Peter ZijlstraDoing RCU-idle outside the driver, only to then temporarily enable it again before going idle is daft.
Notably the cpu_pm_*() calls implicitly re-enable RCU for a bit.
--- drivers/cpuidle/cpuidle-mvebu-v7.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/cpuidle/cpuidle-mvebu-v7.c +7 -0
@@ -36,7 +36,10 @@ static int mvebu_v7_enter_idle(struct cpif (drv->states[index].flags &MVEBU_V7_FLAG_DEEP_IDLE ) deepidle = true; +ct_idle_enter (); ret =mvebu_v7_cpu_suspend (deepidle); +ct_idle_exit (); +cpu_pm_exit ();if (ret)
@@ -49,6 +52,7 @@ static struct cpuidle_driver armadaxp_id .name = "armada_xp_idle", .states[0] = ARM_CPUIDLE_WFI_STATE, .states[1] = { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = mvebu_v7_enter_idle, .exit_latency = 100, .power_usage = 50,
@@ -57,6 +61,7 @@ static struct cpuidle_driver armadaxp_id .desc = "CPU power down", }, .states[2] = { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = mvebu_v7_enter_idle, .exit_latency = 1000, .power_usage = 5,
@@ -72,6 +77,7 @@ static struct cpuidle_driver armada370_i .name = "armada_370_idle", .states[0] = ARM_CPUIDLE_WFI_STATE, .states[1] = { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = mvebu_v7_enter_idle, .exit_latency = 100, .power_usage = 5,
@@ -87,6 +93,7 @@ static struct cpuidle_driver armada38x_i .name = "armada_38x_idle", .states[0] = ARM_CPUIDLE_WFI_STATE, .states[1] = { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = mvebu_v7_enter_idle, .exit_latency = 10, .power_usage = 5,
[tip: sched/core] cpuidle, armada: Push RCU-idle into driver
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4ce40e9dbe83153f60d7e4ccd24a1eb4f8264f6a Gitweb: https://git.kernel.org/tip/4ce40e9dbe83153f60d7e4ccd24a1eb4f8264f6a Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:24 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:47:27 +01:00
cpuidle, armada: Push RCU-idle into driver
Doing RCU-idle outside the driver, only to then temporarily enable it again before going idle is suboptimal.
Notably the cpu_pm_*() calls implicitly re-enable RCU for a bit.
--- drivers/cpuidle/cpuidle-mvebu-v7.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/cpuidle/cpuidle-mvebu-v7.c +7 -0
@@ -36,7 +36,10 @@ static int mvebu_v7_enter_idle(struct cpuidle_device *dev,if (drv->states[index].flags &MVEBU_V7_FLAG_DEEP_IDLE ) deepidle = true; +ct_idle_enter (); ret =mvebu_v7_cpu_suspend (deepidle); +ct_idle_exit (); +cpu_pm_exit ();if (ret)
@@ -49,6 +52,7 @@ static struct cpuidle_driver armadaxp_idle_driver = { .name = "armada_xp_idle", .states[0] = ARM_CPUIDLE_WFI_STATE, .states[1] = { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = mvebu_v7_enter_idle, .exit_latency = 100, .power_usage = 50,
@@ -57,6 +61,7 @@ static struct cpuidle_driver armadaxp_idle_driver = { .desc = "CPU power down", }, .states[2] = { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = mvebu_v7_enter_idle, .exit_latency = 1000, .power_usage = 5,
@@ -72,6 +77,7 @@ static struct cpuidle_driver armada370_idle_driver = { .name = "armada_370_idle", .states[0] = ARM_CPUIDLE_WFI_STATE, .states[1] = { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = mvebu_v7_enter_idle, .exit_latency = 100, .power_usage = 5,
@@ -87,6 +93,7 @@ static struct cpuidle_driver armada38x_idle_driver = { .name = "armada_38x_idle", .states[0] = ARM_CPUIDLE_WFI_STATE, .states[1] = { + .flags = CPUIDLE_FLAG_RCU_IDLE, .enter = mvebu_v7_enter_idle, .exit_latency = 10, .power_usage = 5,
Peter Zijlstra
maintainerPATCH v3 11/51 cpuidle,omap4: Push RCU-idle into driver
In reply to Peter ZijlstraDoing RCU-idle outside the driver, only to then temporarily enable it again, some *four* times, before going idle is daft.
Notably three times explicitly using RCU_NONIDLE() and once implicitly through cpu_pm_*().
--- arch/arm/mach-omap2/cpuidle44xx.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-)
arch/arm/mach-omap2/cpuidle44xx.c +18 -11
@@ -105,7 +105,9 @@ static int omap_enter_idle_smp(struct cp }raw_spin_unlock_irqrestore (&mpu_lock, flag); +ct_idle_enter ();omap4_enter_lowpower (dev->cpu, cx->cpu_state); +ct_idle_exit ();raw_spin_lock_irqsave (&mpu_lock, flag);if (cx->mpu_state_vote ==num_online_cpus ())
@@ -151,10 +153,10 @@ static int omap_enter_idle_coupled(struc (cx->mpu_logic_state ==PWRDM_POWER_OFF );/* Enter broadcast mode for periodic timers */ -RCU_NONIDLE (tick_broadcast_enable ()); +tick_broadcast_enable ();/* Enter broadcast mode for one-shot timers */ -RCU_NONIDLE (tick_broadcast_enter ()); +tick_broadcast_enter ();/* * Call idle CPU PM enter notifier chain so that
@@ -166,7 +168,7 @@ static int omap_enter_idle_coupled(strucif (dev->cpu ==0 ) {pwrdm_set_logic_retst (mpu_pd, cx->mpu_logic_state); -RCU_NONIDLE (omap_set_pwrdm_state (mpu_pd, cx->mpu_state)); +omap_set_pwrdm_state (mpu_pd, cx->mpu_state);/* * Call idle CPU cluster PM enter notifier chain
@@ -178,14 +180,16 @@ static int omap_enter_idle_coupled(struc index =0 ; cx = state_ptr + index;pwrdm_set_logic_retst (mpu_pd, cx->mpu_logic_state); -RCU_NONIDLE (omap_set_pwrdm_state (mpu_pd, cx->mpu_state)); +omap_set_pwrdm_state (mpu_pd, cx->mpu_state); mpuss_can_lose_context =0 ; } } } +ct_idle_enter ();omap4_enter_lowpower (dev->cpu, cx->cpu_state); cpu_done[dev->cpu] = true; +ct_idle_exit ();/* Wakeup CPU1 only if it is not offlined */ if (dev->cpu ==0 &&cpumask_test_cpu (1 , cpu_online_mask)) {
@@ -194,9 +198,9 @@ static int omap_enter_idle_coupled(struc mpuss_can_lose_context)gic_dist_disable (); -RCU_NONIDLE (clkdm_deny_idle (cpu_clkdm[1 ])); -RCU_NONIDLE (omap_set_pwrdm_state (cpu_pd[1 ],PWRDM_POWER_ON )); -RCU_NONIDLE (clkdm_allow_idle (cpu_clkdm[1 ])); +clkdm_deny_idle (cpu_clkdm[1 ]); +omap_set_pwrdm_state (cpu_pd[1 ],PWRDM_POWER_ON ); +clkdm_allow_idle (cpu_clkdm[1 ]);if (IS_PM44XX_ERRATUM (PM_OMAP4_ROM_SMP_BOOT_ERRATUM_GICD ) && mpuss_can_lose_context) {
@@ -222,7 +226,7 @@ static int omap_enter_idle_coupled(struccpu_pm_exit (); cpu_pm_out: -RCU_NONIDLE (tick_broadcast_exit ()); +tick_broadcast_exit (); fail:cpuidle_coupled_parallel_barrier (dev, &abort_barrier);
@@ -247,7 +251,8 @@ static struct cpuidle_driver omap4_idle_/* C2 - CPU0 OFF + CPU1 OFF + MPU CSWR */ .exit_latency = 328 + 440, .target_residency = 960, - .flags = CPUIDLE_FLAG_COUPLED, + .flags =CPUIDLE_FLAG_COUPLED | + CPUIDLE_FLAG_RCU_IDLE, .enter = omap_enter_idle_coupled, .name = "C2", .desc = "CPUx OFF, MPUSS CSWR",
@@ -256,7 +261,8 @@ static struct cpuidle_driver omap4_idle_/* C3 - CPU0 OFF + CPU1 OFF + MPU OSWR */ .exit_latency = 460 + 518, .target_residency = 1100, - .flags = CPUIDLE_FLAG_COUPLED, + .flags =CPUIDLE_FLAG_COUPLED | + CPUIDLE_FLAG_RCU_IDLE, .enter = omap_enter_idle_coupled, .name = "C3", .desc = "CPUx OFF, MPUSS OSWR",
@@ -282,7 +288,8 @@ static struct cpuidle_driver omap5_idle_/* C2 - CPU0 RET + CPU1 RET + MPU CSWR */ .exit_latency = 48 + 60, .target_residency = 100, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .enter = omap_enter_idle_smp, .name = "C2", .desc = "CPUx CSWR, MPUSS CSWR",
[tip: sched/core] cpuidle, OMAP4: Push RCU-idle into driver
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: c3d42418dca53d6c498a48c408f7a45289593650 Gitweb: https://git.kernel.org/tip/c3d42418dca53d6c498a48c408f7a45289593650 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:25 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:47:49 +01:00
cpuidle, OMAP4: Push RCU-idle into driver
Doing RCU-idle outside the driver, only to then temporarily enable it again, some *four* times, before going idle is suboptimal.
Notably three times explicitly using RCU_NONIDLE() and once implicitly through cpu_pm_*().
--- arch/arm/mach-omap2/cpuidle44xx.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-)
arch/arm/mach-omap2/cpuidle44xx.c +18 -11
@@ -105,7 +105,9 @@ static int omap_enter_idle_smp(struct cpuidle_device *dev, }raw_spin_unlock_irqrestore (&mpu_lock, flag); +ct_idle_enter ();omap4_enter_lowpower (dev->cpu, cx->cpu_state); +ct_idle_exit ();raw_spin_lock_irqsave (&mpu_lock, flag);if (cx->mpu_state_vote ==num_online_cpus ())
@@ -151,10 +153,10 @@ static int omap_enter_idle_coupled(struct cpuidle_device *dev, (cx->mpu_logic_state ==PWRDM_POWER_OFF );/* Enter broadcast mode for periodic timers */ -RCU_NONIDLE (tick_broadcast_enable ()); +tick_broadcast_enable ();/* Enter broadcast mode for one-shot timers */ -RCU_NONIDLE (tick_broadcast_enter ()); +tick_broadcast_enter ();/* * Call idle CPU PM enter notifier chain so that
@@ -166,7 +168,7 @@ static int omap_enter_idle_coupled(struct cpuidle_device *dev,if (dev->cpu ==0 ) {pwrdm_set_logic_retst (mpu_pd, cx->mpu_logic_state); -RCU_NONIDLE (omap_set_pwrdm_state (mpu_pd, cx->mpu_state)); +omap_set_pwrdm_state (mpu_pd, cx->mpu_state);/* * Call idle CPU cluster PM enter notifier chain
@@ -178,14 +180,16 @@ static int omap_enter_idle_coupled(struct cpuidle_device *dev, index =0 ; cx = state_ptr + index;pwrdm_set_logic_retst (mpu_pd, cx->mpu_logic_state); -RCU_NONIDLE (omap_set_pwrdm_state (mpu_pd, cx->mpu_state)); +omap_set_pwrdm_state (mpu_pd, cx->mpu_state); mpuss_can_lose_context =0 ; } } } +ct_idle_enter ();omap4_enter_lowpower (dev->cpu, cx->cpu_state); cpu_done[dev->cpu] = true; +ct_idle_exit ();/* Wakeup CPU1 only if it is not offlined */ if (dev->cpu ==0 &&cpumask_test_cpu (1 , cpu_online_mask)) {
@@ -194,9 +198,9 @@ static int omap_enter_idle_coupled(struct cpuidle_device *dev, mpuss_can_lose_context)gic_dist_disable (); -RCU_NONIDLE (clkdm_deny_idle (cpu_clkdm[1 ])); -RCU_NONIDLE (omap_set_pwrdm_state (cpu_pd[1 ],PWRDM_POWER_ON )); -RCU_NONIDLE (clkdm_allow_idle (cpu_clkdm[1 ])); +clkdm_deny_idle (cpu_clkdm[1 ]); +omap_set_pwrdm_state (cpu_pd[1 ],PWRDM_POWER_ON ); +clkdm_allow_idle (cpu_clkdm[1 ]);if (IS_PM44XX_ERRATUM (PM_OMAP4_ROM_SMP_BOOT_ERRATUM_GICD ) && mpuss_can_lose_context) {
@@ -222,7 +226,7 @@ static int omap_enter_idle_coupled(struct cpuidle_device *dev,cpu_pm_exit (); cpu_pm_out: -RCU_NONIDLE (tick_broadcast_exit ()); +tick_broadcast_exit (); fail:cpuidle_coupled_parallel_barrier (dev, &abort_barrier);
@@ -247,7 +251,8 @@ static struct cpuidle_driver omap4_idle_driver = {/* C2 - CPU0 OFF + CPU1 OFF + MPU CSWR */ .exit_latency = 328 + 440, .target_residency = 960, - .flags = CPUIDLE_FLAG_COUPLED, + .flags =CPUIDLE_FLAG_COUPLED | + CPUIDLE_FLAG_RCU_IDLE, .enter = omap_enter_idle_coupled, .name = "C2", .desc = "CPUx OFF, MPUSS CSWR",
@@ -256,7 +261,8 @@ static struct cpuidle_driver omap4_idle_driver = {/* C3 - CPU0 OFF + CPU1 OFF + MPU OSWR */ .exit_latency = 460 + 518, .target_residency = 1100, - .flags = CPUIDLE_FLAG_COUPLED, + .flags =CPUIDLE_FLAG_COUPLED | + CPUIDLE_FLAG_RCU_IDLE, .enter = omap_enter_idle_coupled, .name = "C3", .desc = "CPUx OFF, MPUSS OSWR",
@@ -282,7 +288,8 @@ static struct cpuidle_driver omap5_idle_driver = {/* C2 - CPU0 RET + CPU1 RET + MPU CSWR */ .exit_latency = 48 + 60, .target_residency = 100, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .enter = omap_enter_idle_smp, .name = "C2", .desc = "CPUx CSWR, MPUSS CSWR",
Peter Zijlstra
maintainerPATCH v3 12/51 cpuidle,dt: Push RCU-idle into driver
In reply to Peter ZijlstraDoing RCU-idle outside the driver, only to then temporarily enable it again before going idle is daft.
Notably: this converts all dt_init_idle_driver() and __CPU_PM_CPU_IDLE_ENTER() users for they are inextrably intertwined.
--- drivers/acpi/processor_idle.c | 2 ++ drivers/cpuidle/cpuidle-arm.c | 1 + drivers/cpuidle/cpuidle-big_little.c | 8 ++++++-- drivers/cpuidle/cpuidle-psci.c | 1 + drivers/cpuidle/cpuidle-qcom-spm.c | 1 + drivers/cpuidle/cpuidle-riscv-sbi.c | 1 + drivers/cpuidle/dt_idle_states.c | 2 +- include/linux/cpuidle.h | 2 ++ 8 files changed, 15 insertions(+), 3 deletions(-)
drivers/acpi/processor_idle.c +2 -0
@@ -1219,6 +1219,8 @@ static int acpi_processor_setup_lpi_stat state->target_residency = lpi->min_residency;if (lpi->arch_flags) state->flags |=CPUIDLE_FLAG_TIMER_STOP ; +if (i !=0 && lpi->entry_method ==ACPI_CSTATE_FFH ) + state->flags |=CPUIDLE_FLAG_RCU_IDLE ; state->enter = acpi_idle_lpi_enter; drv->safe_state_index = i; }
drivers/cpuidle/cpuidle-big_little.c +6 -2
@@ -64,7 +64,8 @@ static struct cpuidle_driver bl_idle_lit .enter = bl_enter_powerdown, .exit_latency = 700, .target_residency = 2500, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .name = "C1", .desc = "ARM little-cluster power down", },
@@ -85,7 +86,8 @@ static struct cpuidle_driver bl_idle_big .enter = bl_enter_powerdown, .exit_latency = 500, .target_residency = 2000, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .name = "C1", .desc = "ARM big-cluster power down", },
@@ -124,11 +126,13 @@ static int bl_enter_powerdown(struct cpu struct cpuidle_driver *drv, int idx) {cpu_pm_enter (); +ct_idle_enter ();cpu_suspend (0 , bl_powerdown_finisher);/* signals the MCPM core that CPU is out of low power state */ mcpm_cpu_powered_up (); +ct_idle_exit ();cpu_pm_exit ();
drivers/cpuidle/dt_idle_states.c +1 -1
@@ -77,7 +77,7 @@ static int init_state_node(struct cpuidlif (err) desc = state_node->name; - idle_state->flags =0 ; + idle_state->flags =CPUIDLE_FLAG_RCU_IDLE ;if (of_property_read_bool (state_node,"local-timer-stop" )) idle_state->flags |=CPUIDLE_FLAG_TIMER_STOP ;/*
include/linux/cpuidle.h +2 -0
@@ -289,7 +289,9 @@ extern s64 cpuidle_governor_latency_req(if (!is_retention) \ __ret =cpu_pm_enter (); \if (!__ret) { \ +ct_idle_enter (); \ __ret =low_level_idle_enter (state); \ +ct_idle_exit (); \if (!is_retention) \cpu_pm_exit (); \ } \
[tip: sched/core] cpuidle, dt: Push RCU-idle into driver
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 0c5ffc3d7b15978c6b184938cd6b8af06e436424 Gitweb: https://git.kernel.org/tip/0c5ffc3d7b15978c6b184938cd6b8af06e436424 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:26 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:00 +01:00
cpuidle, dt: Push RCU-idle into driver
Doing RCU-idle outside the driver, only to then temporarily enable it again before going idle is suboptimal.
Notably: this converts all dt_init_idle_driver() and __CPU_PM_CPU_IDLE_ENTER() users for they are inextrably intertwined.
--- drivers/acpi/processor_idle.c | 2 ++ drivers/cpuidle/cpuidle-big_little.c | 8 ++++++-- drivers/cpuidle/dt_idle_states.c | 2 +- include/linux/cpuidle.h | 2 ++ 4 files changed, 11 insertions(+), 3 deletions(-)
drivers/acpi/processor_idle.c +2 -0
@@ -1219,6 +1219,8 @@ static int acpi_processor_setup_lpi_states(struct acpi_processor *pr) state->target_residency = lpi->min_residency;if (lpi->arch_flags) state->flags |=CPUIDLE_FLAG_TIMER_STOP ; +if (i !=0 && lpi->entry_method ==ACPI_CSTATE_FFH ) + state->flags |=CPUIDLE_FLAG_RCU_IDLE ; state->enter = acpi_idle_lpi_enter; drv->safe_state_index = i; }
drivers/cpuidle/cpuidle-big_little.c +6 -2
@@ -64,7 +64,8 @@ static struct cpuidle_driver bl_idle_little_driver = { .enter = bl_enter_powerdown, .exit_latency = 700, .target_residency = 2500, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .name = "C1", .desc = "ARM little-cluster power down", },
@@ -85,7 +86,8 @@ static struct cpuidle_driver bl_idle_big_driver = { .enter = bl_enter_powerdown, .exit_latency = 500, .target_residency = 2000, - .flags = CPUIDLE_FLAG_TIMER_STOP, + .flags =CPUIDLE_FLAG_TIMER_STOP | + CPUIDLE_FLAG_RCU_IDLE, .name = "C1", .desc = "ARM big-cluster power down", },
@@ -124,11 +126,13 @@ static int bl_enter_powerdown(struct cpuidle_device *dev, struct cpuidle_driver *drv, int idx) {cpu_pm_enter (); +ct_idle_enter ();cpu_suspend (0 , bl_powerdown_finisher);/* signals the MCPM core that CPU is out of low power state */ mcpm_cpu_powered_up (); +ct_idle_exit ();cpu_pm_exit ();
drivers/cpuidle/dt_idle_states.c +1 -1
@@ -77,7 +77,7 @@ static int init_state_node(struct cpuidle_state *idle_state,if (err) desc = state_node->name; - idle_state->flags =0 ; + idle_state->flags =CPUIDLE_FLAG_RCU_IDLE ;if (of_property_read_bool (state_node,"local-timer-stop" )) idle_state->flags |=CPUIDLE_FLAG_TIMER_STOP ;/*
include/linux/cpuidle.h +2 -0
@@ -289,7 +289,9 @@ extern s64 cpuidle_governor_latency_req(unsigned int cpu);if (!is_retention) \ __ret =cpu_pm_enter (); \if (!__ret) { \ +ct_idle_enter (); \ __ret =low_level_idle_enter (state); \ +ct_idle_exit (); \if (!is_retention) \cpu_pm_exit (); \ } \
Peter Zijlstra
maintainerPATCH v3 13/51 cpuidle: Fix ct_idle_*() usage
In reply to Peter ZijlstraThe whole disable-RCU, enable-IRQS dance is very intricate since changing IRQ state is traced, which depends on RCU.
Add two helpers for the cpuidle case that mirror the entry code.
--- arch/arm/mach-imx/cpuidle-imx6q.c | 4 +-- arch/arm/mach-imx/cpuidle-imx6sx.c | 4 +-- arch/arm/mach-omap2/cpuidle34xx.c | 4 +-- arch/arm/mach-omap2/cpuidle44xx.c | 8 +++--- drivers/acpi/processor_idle.c | 8 ++++-- drivers/cpuidle/cpuidle-big_little.c | 4 +-- drivers/cpuidle/cpuidle-mvebu-v7.c | 4 +-- drivers/cpuidle/cpuidle-psci.c | 4 +-- drivers/cpuidle/cpuidle-riscv-sbi.c | 4 +-- drivers/cpuidle/cpuidle-tegra.c | 8 +++--- drivers/cpuidle/cpuidle.c | 11 ++++---- include/linux/clockchips.h | 4 +-- include/linux/cpuidle.h | 34 ++++++++++++++++++++++++-- kernel/sched/idle.c | 45 ++++++++++------------------------- kernel/time/tick-broadcast.c | 6 +++- 15 files changed, 86 insertions(+), 66 deletions(-)
arch/arm/mach-imx/cpuidle-imx6q.c +2 -2
@@ -25,9 +25,9 @@ static int imx6q_enter_wait(struct cpuidimx6_set_lpm (WAIT_UNCLOCKED );raw_spin_unlock (&cpuidle_lock); -ct_idle_enter (); +ct_cpuidle_enter ();cpu_do_idle (); -ct_idle_exit (); +ct_cpuidle_exit ();raw_spin_lock (&cpuidle_lock);if (num_idle_cpus-- ==num_online_cpus ())
arch/arm/mach-imx/cpuidle-imx6sx.c +2 -2
@@ -47,9 +47,9 @@ static int imx6sx_enter_wait(struct cpuicpu_pm_enter ();cpu_cluster_pm_enter (); -ct_idle_enter (); +ct_cpuidle_enter ();cpu_suspend (0 , imx6sx_idle_finish); -ct_idle_exit (); +ct_cpuidle_exit ();cpu_cluster_pm_exit ();cpu_pm_exit ();
arch/arm/mach-omap2/cpuidle34xx.c +2 -2
@@ -133,9 +133,9 @@ static int omap3_enter_idle(struct cpuid }/* Execute ARM wfi */ -ct_idle_enter (); +ct_cpuidle_enter ();omap_sram_idle (); -ct_idle_exit (); +ct_cpuidle_exit ();/* * Call idle CPU PM enter notifier chain to restore
arch/arm/mach-omap2/cpuidle44xx.c +4 -4
@@ -105,9 +105,9 @@ static int omap_enter_idle_smp(struct cp }raw_spin_unlock_irqrestore (&mpu_lock, flag); -ct_idle_enter (); +ct_cpuidle_enter ();omap4_enter_lowpower (dev->cpu, cx->cpu_state); -ct_idle_exit (); +ct_cpuidle_exit ();raw_spin_lock_irqsave (&mpu_lock, flag);if (cx->mpu_state_vote ==num_online_cpus ())
@@ -186,10 +186,10 @@ static int omap_enter_idle_coupled(struc } } -ct_idle_enter (); +ct_cpuidle_enter ();omap4_enter_lowpower (dev->cpu, cx->cpu_state); cpu_done[dev->cpu] = true; -ct_idle_exit (); +ct_cpuidle_exit ();/* Wakeup CPU1 only if it is not offlined */ if (dev->cpu ==0 &&cpumask_test_cpu (1 , cpu_online_mask)) {
drivers/acpi/processor_idle.c +6 -2
@@ -642,6 +642,8 @@ static int __cpuidle acpi_idle_enter_bm(*/ bool dis_bm = pr->flags.bm_control; +instrumentation_begin (); +/* If we can skip BM, demote to a safe state. */ if (!cx->bm_sts_skip &&acpi_idle_bm_check ()) { dis_bm = false;
@@ -663,11 +665,11 @@ static int __cpuidle acpi_idle_enter_bm(raw_spin_unlock (&c3_lock); } -ct_idle_enter (); +ct_cpuidle_enter ();acpi_idle_do_entry (cx); -ct_idle_exit (); +ct_cpuidle_exit ();/* Re-enable bus master arbitration */ if (dis_bm) {
@@ -677,6 +679,8 @@ static int __cpuidle acpi_idle_enter_bm(raw_spin_unlock (&c3_lock); } +instrumentation_end (); +return index; }
drivers/cpuidle/cpuidle-big_little.c +2 -2
@@ -126,13 +126,13 @@ static int bl_enter_powerdown(struct cpu struct cpuidle_driver *drv, int idx) {cpu_pm_enter (); -ct_idle_enter (); +ct_cpuidle_enter ();cpu_suspend (0 , bl_powerdown_finisher);/* signals the MCPM core that CPU is out of low power state */ mcpm_cpu_powered_up (); -ct_idle_exit (); +ct_cpuidle_exit ();cpu_pm_exit ();
drivers/cpuidle/cpuidle-mvebu-v7.c +2 -2
@@ -36,9 +36,9 @@ static int mvebu_v7_enter_idle(struct cpif (drv->states[index].flags &MVEBU_V7_FLAG_DEEP_IDLE ) deepidle = true; -ct_idle_enter (); +ct_cpuidle_enter (); ret =mvebu_v7_cpu_suspend (deepidle); -ct_idle_exit (); +ct_cpuidle_exit ();cpu_pm_exit ();
drivers/cpuidle/cpuidle-psci.c +2 -2
@@ -74,7 +74,7 @@ static int __psci_enter_domain_idle_statelse pm_runtime_put_sync_suspend (pd_dev); -ct_idle_enter (); +ct_cpuidle_enter (); state =psci_get_domain_state ();if (!state)
@@ -82,7 +82,7 @@ static int __psci_enter_domain_idle_stat ret =psci_cpu_suspend_enter (state) ?-1 : idx; -ct_idle_exit (); +ct_cpuidle_exit ();if (s2idle)dev_pm_genpd_resume (pd_dev);
drivers/cpuidle/cpuidle-riscv-sbi.c +2 -2
@@ -126,7 +126,7 @@ static int __sbi_enter_domain_idle_stateelse pm_runtime_put_sync_suspend (pd_dev); -ct_idle_enter (); +ct_cpuidle_enter ();if (sbi_is_domain_state_available ()) state =sbi_get_domain_state ();
@@ -135,7 +135,7 @@ static int __sbi_enter_domain_idle_state ret =sbi_suspend (state) ?-1 : idx; -ct_idle_exit (); +ct_cpuidle_exit ();if (s2idle)dev_pm_genpd_resume (pd_dev);
drivers/cpuidle/cpuidle-tegra.c +4 -4
@@ -183,7 +183,7 @@ static int tegra_cpuidle_state_enter(strtegra_pm_set_cpu_in_lp2 ();cpu_pm_enter (); -ct_idle_enter (); +ct_cpuidle_enter ();switch (index) {case TEGRA_C7 :
@@ -199,7 +199,7 @@ static int tegra_cpuidle_state_enter(strbreak ; } -ct_idle_exit (); +ct_cpuidle_exit ();cpu_pm_exit ();tegra_pm_clear_cpu_in_lp2 ();
@@ -240,10 +240,10 @@ static int tegra_cpuidle_enter(struct cpif (index ==TEGRA_C1 ) {if (do_rcu) -ct_idle_enter (); +ct_cpuidle_enter (); ret =arm_cpuidle_simple_enter (dev, drv, index);if (do_rcu) -ct_idle_exit (); +ct_cpuidle_exit (); }else ret =tegra_cpuidle_state_enter (dev, index, cpu);
drivers/cpuidle/cpuidle.c +6 -5
@@ -14,6 +14,7 @@#include <linux/mutex.h> #include <linux/sched.h> #include <linux/sched/clock.h> +#include <linux/sched/idle.h> #include <linux/notifier.h> #include <linux/pm_qos.h> #include <linux/cpu.h>
@@ -152,12 +153,12 @@ static void enter_s2idle_proper(struct c*/ stop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -ct_idle_enter (); +ct_cpuidle_enter (); target_state->enter_s2idle (dev, drv, index);if (WARN_ON_ONCE (!irqs_disabled ())) -local_irq_disable (); +raw_local_irq_disable ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -ct_idle_exit (); +ct_cpuidle_exit ();tick_unfreeze ();start_critical_timings ();
@@ -235,14 +236,14 @@ int cpuidle_enter_state(struct cpuidle_dstop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -ct_idle_enter (); +ct_cpuidle_enter (); entered_state = target_state->enter (dev, drv, index);if (WARN_ONCE (!irqs_disabled (),"%ps leaked IRQ state" , target_state->enter))raw_local_irq_disable ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -ct_idle_exit (); +ct_cpuidle_exit ();start_critical_timings ();sched_clock_idle_wakeup_event ();
include/linux/clockchips.h +2 -2
@@ -211,7 +211,7 @@ extern int tick_receive_broadcast(void);extern void tick_setup_hrtimer_broadcast (void );extern int tick_check_broadcast_expired (void );# else -static inline int tick_check_broadcast_expired (void ) {return 0 ; } +static __always_inline int tick_check_broadcast_expired(void) { return 0; }static inline void tick_setup_hrtimer_broadcast (void ) { }# endif
@@ -219,7 +219,7 @@ static inline void tick_setup_hrtimer_brstatic inline void clockevents_suspend (void ) { }static inline void clockevents_resume (void ) { } -static inline int tick_check_broadcast_expired (void ) {return 0 ; } +static __always_inline int tick_check_broadcast_expired(void) { return 0; }static inline void tick_setup_hrtimer_broadcast (void ) { }#endif /* !CONFIG_GENERIC_CLOCKEVENTS */
include/linux/cpuidle.h +32 -2
@@ -14,6 +14,7 @@#include <linux/percpu.h> #include <linux/list.h> #include <linux/hrtimer.h> +#include <linux/context_tracking.h> #define CPUIDLE_STATE_MAX 10#define CPUIDLE_NAME_LEN 16
@@ -115,6 +116,35 @@ struct cpuidle_device { DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices); DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev); +static __always_inline void ct_cpuidle_enter(void) +{ +lockdep_assert_irqs_disabled (); +/* +* Idle is allowed to (temporary) enable IRQs. It +* will return with IRQs disabled. +* +* Trace IRQs enable here, then switch off RCU, and have +* arch_cpu_idle() use raw_local_irq_enable(). Note that +* ct_idle_enter() relies on lockdep IRQ state, so switch that +* last -- this is very similar to the entry code. +*/ +trace_hardirqs_on_prepare (); +lockdep_hardirqs_on_prepare (); +instrumentation_end (); +ct_idle_enter (); +lockdep_hardirqs_on (_RET_IP_); +} + +static __always_inline void ct_cpuidle_exit(void) +{ +/* +* Carefully undo the above. +*/ +lockdep_hardirqs_off (_RET_IP_); +ct_idle_exit (); +instrumentation_begin (); +} +/**************************** * CPUIDLE DRIVER INTERFACE * ****************************/
@@ -289,9 +319,9 @@ extern s64 cpuidle_governor_latency_req(if (!is_retention) \ __ret =cpu_pm_enter (); \if (!__ret) { \ -ct_idle_enter (); \ +ct_cpuidle_enter (); \ __ret =low_level_idle_enter (state); \ -ct_idle_exit (); \ +ct_cpuidle_exit (); \if (!is_retention) \cpu_pm_exit (); \ } \
kernel/sched/idle.c +13 -32
75 lines not on this page · raw message
kernel/time/tick-broadcast.c +5 -1
@@ -622,9 +622,13 @@ struct cpumask *tick_get_broadcast_onesh* to avoid a deep idle transition as we are about to get the * broadcast IPI right away. */ -int tick_check_broadcast_expired (void ) +noinstr int tick_check_broadcast_expired(void) { +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H +return arch_test_bit (smp_processor_id (),cpumask_bits (tick_broadcast_force_mask)); +#else return cpumask_test_cpu (smp_processor_id (), tick_broadcast_force_mask); +#endif }/*
[tip: sched/core] cpuidle: Fix ct_idle_*() usage
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: a01353cf1896ea5b8a7bbc5e2b2d38feed8b7aaa Gitweb: https://git.kernel.org/tip/a01353cf1896ea5b8a7bbc5e2b2d38feed8b7aaa Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:27 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
cpuidle: Fix ct_idle_*() usage
The whole disable-RCU, enable-IRQS dance is very intricate since changing IRQ state is traced, which depends on RCU.
Add two helpers for the cpuidle case that mirror the entry code:
ct_cpuidle_enter()
ct_cpuidle_exit()
And fix all the cases where the enter/exit dance was buggy.
--- arch/arm/mach-imx/cpuidle-imx6q.c | 4 +- arch/arm/mach-imx/cpuidle-imx6sx.c | 4 +- arch/arm/mach-omap2/cpuidle34xx.c | 4 +- arch/arm/mach-omap2/cpuidle44xx.c | 8 ++--- drivers/acpi/processor_idle.c | 8 +++-- drivers/cpuidle/cpuidle-big_little.c | 4 +- drivers/cpuidle/cpuidle-mvebu-v7.c | 4 +- drivers/cpuidle/cpuidle-psci.c | 4 +- drivers/cpuidle/cpuidle-riscv-sbi.c | 4 +- drivers/cpuidle/cpuidle-tegra.c | 8 ++--- drivers/cpuidle/cpuidle.c | 11 +++---- include/linux/clockchips.h | 4 +- include/linux/cpuidle.h | 34 ++++++++++++++++++-- kernel/sched/idle.c | 45 +++++++-------------------- kernel/time/tick-broadcast.c | 6 +++- 15 files changed, 86 insertions(+), 66 deletions(-)
arch/arm/mach-imx/cpuidle-imx6q.c +2 -2
@@ -25,9 +25,9 @@ static int imx6q_enter_wait(struct cpuidle_device *dev,imx6_set_lpm (WAIT_UNCLOCKED );raw_spin_unlock (&cpuidle_lock); -ct_idle_enter (); +ct_cpuidle_enter ();cpu_do_idle (); -ct_idle_exit (); +ct_cpuidle_exit ();raw_spin_lock (&cpuidle_lock);if (num_idle_cpus-- ==num_online_cpus ())
arch/arm/mach-imx/cpuidle-imx6sx.c +2 -2
@@ -47,9 +47,9 @@ static int imx6sx_enter_wait(struct cpuidle_device *dev,cpu_pm_enter ();cpu_cluster_pm_enter (); -ct_idle_enter (); +ct_cpuidle_enter ();cpu_suspend (0 , imx6sx_idle_finish); -ct_idle_exit (); +ct_cpuidle_exit ();cpu_cluster_pm_exit ();cpu_pm_exit ();
arch/arm/mach-omap2/cpuidle34xx.c +2 -2
@@ -133,9 +133,9 @@ static int omap3_enter_idle(struct cpuidle_device *dev, }/* Execute ARM wfi */ -ct_idle_enter (); +ct_cpuidle_enter ();omap_sram_idle (); -ct_idle_exit (); +ct_cpuidle_exit ();/* * Call idle CPU PM enter notifier chain to restore
arch/arm/mach-omap2/cpuidle44xx.c +4 -4
@@ -105,9 +105,9 @@ static int omap_enter_idle_smp(struct cpuidle_device *dev, }raw_spin_unlock_irqrestore (&mpu_lock, flag); -ct_idle_enter (); +ct_cpuidle_enter ();omap4_enter_lowpower (dev->cpu, cx->cpu_state); -ct_idle_exit (); +ct_cpuidle_exit ();raw_spin_lock_irqsave (&mpu_lock, flag);if (cx->mpu_state_vote ==num_online_cpus ())
@@ -186,10 +186,10 @@ static int omap_enter_idle_coupled(struct cpuidle_device *dev, } } -ct_idle_enter (); +ct_cpuidle_enter ();omap4_enter_lowpower (dev->cpu, cx->cpu_state); cpu_done[dev->cpu] = true; -ct_idle_exit (); +ct_cpuidle_exit ();/* Wakeup CPU1 only if it is not offlined */ if (dev->cpu ==0 &&cpumask_test_cpu (1 , cpu_online_mask)) {
drivers/acpi/processor_idle.c +6 -2
@@ -642,6 +642,8 @@ static int __cpuidle acpi_idle_enter_bm(struct cpuidle_driver *drv,*/ bool dis_bm = pr->flags.bm_control; +instrumentation_begin (); +/* If we can skip BM, demote to a safe state. */ if (!cx->bm_sts_skip &&acpi_idle_bm_check ()) { dis_bm = false;
@@ -663,11 +665,11 @@ static int __cpuidle acpi_idle_enter_bm(struct cpuidle_driver *drv,raw_spin_unlock (&c3_lock); } -ct_idle_enter (); +ct_cpuidle_enter ();acpi_idle_do_entry (cx); -ct_idle_exit (); +ct_cpuidle_exit ();/* Re-enable bus master arbitration */ if (dis_bm) {
@@ -677,6 +679,8 @@ static int __cpuidle acpi_idle_enter_bm(struct cpuidle_driver *drv,raw_spin_unlock (&c3_lock); } +instrumentation_end (); +return index; }
drivers/cpuidle/cpuidle-big_little.c +2 -2
@@ -126,13 +126,13 @@ static int bl_enter_powerdown(struct cpuidle_device *dev, struct cpuidle_driver *drv, int idx) {cpu_pm_enter (); -ct_idle_enter (); +ct_cpuidle_enter ();cpu_suspend (0 , bl_powerdown_finisher);/* signals the MCPM core that CPU is out of low power state */ mcpm_cpu_powered_up (); -ct_idle_exit (); +ct_cpuidle_exit ();cpu_pm_exit ();
drivers/cpuidle/cpuidle-mvebu-v7.c +2 -2
@@ -36,9 +36,9 @@ static int mvebu_v7_enter_idle(struct cpuidle_device *dev,if (drv->states[index].flags &MVEBU_V7_FLAG_DEEP_IDLE ) deepidle = true; -ct_idle_enter (); +ct_cpuidle_enter (); ret =mvebu_v7_cpu_suspend (deepidle); -ct_idle_exit (); +ct_cpuidle_exit ();cpu_pm_exit ();
drivers/cpuidle/cpuidle-psci.c +2 -2
@@ -74,7 +74,7 @@ static int __psci_enter_domain_idle_state(struct cpuidle_device *dev,else pm_runtime_put_sync_suspend (pd_dev); -ct_idle_enter (); +ct_cpuidle_enter (); state =psci_get_domain_state ();if (!state)
@@ -82,7 +82,7 @@ static int __psci_enter_domain_idle_state(struct cpuidle_device *dev, ret =psci_cpu_suspend_enter (state) ?-1 : idx; -ct_idle_exit (); +ct_cpuidle_exit ();if (s2idle)dev_pm_genpd_resume (pd_dev);
drivers/cpuidle/cpuidle-riscv-sbi.c +2 -2
@@ -126,7 +126,7 @@ static int __sbi_enter_domain_idle_state(struct cpuidle_device *dev,else pm_runtime_put_sync_suspend (pd_dev); -ct_idle_enter (); +ct_cpuidle_enter ();if (sbi_is_domain_state_available ()) state =sbi_get_domain_state ();
@@ -135,7 +135,7 @@ static int __sbi_enter_domain_idle_state(struct cpuidle_device *dev, ret =sbi_suspend (state) ?-1 : idx; -ct_idle_exit (); +ct_cpuidle_exit ();if (s2idle)dev_pm_genpd_resume (pd_dev);
drivers/cpuidle/cpuidle-tegra.c +4 -4
@@ -183,7 +183,7 @@ static int tegra_cpuidle_state_enter(struct cpuidle_device *dev,tegra_pm_set_cpu_in_lp2 ();cpu_pm_enter (); -ct_idle_enter (); +ct_cpuidle_enter ();switch (index) {case TEGRA_C7 :
@@ -199,7 +199,7 @@ static int tegra_cpuidle_state_enter(struct cpuidle_device *dev,break ; } -ct_idle_exit (); +ct_cpuidle_exit ();cpu_pm_exit ();tegra_pm_clear_cpu_in_lp2 ();
@@ -240,10 +240,10 @@ static int tegra_cpuidle_enter(struct cpuidle_device *dev,if (index ==TEGRA_C1 ) {if (do_rcu) -ct_idle_enter (); +ct_cpuidle_enter (); ret =arm_cpuidle_simple_enter (dev, drv, index);if (do_rcu) -ct_idle_exit (); +ct_cpuidle_exit (); }else ret =tegra_cpuidle_state_enter (dev, index, cpu);
drivers/cpuidle/cpuidle.c +6 -5
@@ -14,6 +14,7 @@#include <linux/mutex.h> #include <linux/sched.h> #include <linux/sched/clock.h> +#include <linux/sched/idle.h> #include <linux/notifier.h> #include <linux/pm_qos.h> #include <linux/cpu.h>
@@ -152,12 +153,12 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,*/ stop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -ct_idle_enter (); +ct_cpuidle_enter (); target_state->enter_s2idle (dev, drv, index);if (WARN_ON_ONCE (!irqs_disabled ())) -local_irq_disable (); +raw_local_irq_disable ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -ct_idle_exit (); +ct_cpuidle_exit ();tick_unfreeze ();start_critical_timings ();
@@ -235,14 +236,14 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,stop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -ct_idle_enter (); +ct_cpuidle_enter (); entered_state = target_state->enter (dev, drv, index);if (WARN_ONCE (!irqs_disabled (),"%ps leaked IRQ state" , target_state->enter))raw_local_irq_disable ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -ct_idle_exit (); +ct_cpuidle_exit ();start_critical_timings ();sched_clock_idle_wakeup_event ();
include/linux/clockchips.h +2 -2
@@ -211,7 +211,7 @@ extern int tick_receive_broadcast(void);extern void tick_setup_hrtimer_broadcast (void );extern int tick_check_broadcast_expired (void );# else -static inline int tick_check_broadcast_expired (void ) {return 0 ; } +static __always_inline int tick_check_broadcast_expired(void) { return 0; }static inline void tick_setup_hrtimer_broadcast (void ) { }# endif
@@ -219,7 +219,7 @@ static inline void tick_setup_hrtimer_broadcast(void) { }static inline void clockevents_suspend (void ) { }static inline void clockevents_resume (void ) { } -static inline int tick_check_broadcast_expired (void ) {return 0 ; } +static __always_inline int tick_check_broadcast_expired(void) { return 0; }static inline void tick_setup_hrtimer_broadcast (void ) { }#endif /* !CONFIG_GENERIC_CLOCKEVENTS */
include/linux/cpuidle.h +32 -2
@@ -14,6 +14,7 @@#include <linux/percpu.h> #include <linux/list.h> #include <linux/hrtimer.h> +#include <linux/context_tracking.h> #define CPUIDLE_STATE_MAX 10#define CPUIDLE_NAME_LEN 16
@@ -115,6 +116,35 @@ struct cpuidle_device { DECLARE_PER_CPU(struct cpuidle_device *, cpuidle_devices); DECLARE_PER_CPU(struct cpuidle_device, cpuidle_dev); +static __always_inline void ct_cpuidle_enter(void) +{ +lockdep_assert_irqs_disabled (); +/* +* Idle is allowed to (temporary) enable IRQs. It +* will return with IRQs disabled. +* +* Trace IRQs enable here, then switch off RCU, and have +* arch_cpu_idle() use raw_local_irq_enable(). Note that +* ct_idle_enter() relies on lockdep IRQ state, so switch that +* last -- this is very similar to the entry code. +*/ +trace_hardirqs_on_prepare (); +lockdep_hardirqs_on_prepare (); +instrumentation_end (); +ct_idle_enter (); +lockdep_hardirqs_on (_RET_IP_); +} + +static __always_inline void ct_cpuidle_exit(void) +{ +/* +* Carefully undo the above. +*/ +lockdep_hardirqs_off (_RET_IP_); +ct_idle_exit (); +instrumentation_begin (); +} +/**************************** * CPUIDLE DRIVER INTERFACE * ****************************/
@@ -289,9 +319,9 @@ extern s64 cpuidle_governor_latency_req(unsigned int cpu);if (!is_retention) \ __ret =cpu_pm_enter (); \if (!__ret) { \ -ct_idle_enter (); \ +ct_cpuidle_enter (); \ __ret =low_level_idle_enter (state); \ -ct_idle_exit (); \ +ct_cpuidle_exit (); \if (!is_retention) \cpu_pm_exit (); \ } \
kernel/sched/idle.c +13 -32
75 lines not on this page · raw message
kernel/time/tick-broadcast.c +5 -1
@@ -622,9 +622,13 @@ struct cpumask *tick_get_broadcast_oneshot_mask(void)* to avoid a deep idle transition as we are about to get the * broadcast IPI right away. */ -int tick_check_broadcast_expired (void ) +noinstr int tick_check_broadcast_expired(void) { +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H +return arch_test_bit (smp_processor_id (),cpumask_bits (tick_broadcast_force_mask)); +#else return cpumask_test_cpu (smp_processor_id (), tick_broadcast_force_mask); +#endif }/*
Peter Zijlstra
maintainerPATCH v3 14/51 cpuidle,cpu_pm: Remove RCU fiddling from cpu_pm_{enter,exit}()
In reply to Peter ZijlstraAll callers should still have RCU enabled.
--- kernel/cpu_pm.c | 9 --------- 1 file changed, 9 deletions(-)
kernel/cpu_pm.c +0 -9
@@ -30,16 +30,9 @@ static int cpu_pm_notify(enum cpu_pm_eve {int ret; -/* -* This introduces a RCU read critical section, which could be -* disfunctional in cpu idle. Copy RCU_NONIDLE code to let RCU know -* this. -*/ -ct_irq_enter_irqson ();rcu_read_lock (); ret =raw_notifier_call_chain (&cpu_pm_notifier.chain, event,NULL );rcu_read_unlock (); -ct_irq_exit_irqson ();return notifier_to_errno (ret); }
@@ -49,11 +42,9 @@ static int cpu_pm_notify_robust(enum cpuunsigned long flags;int ret; -ct_irq_enter_irqson ();raw_spin_lock_irqsave (&cpu_pm_notifier.lock, flags); ret =raw_notifier_call_chain_robust (&cpu_pm_notifier.chain, event_up, event_down,NULL );raw_spin_unlock_irqrestore (&cpu_pm_notifier.lock, flags); -ct_irq_exit_irqson ();return notifier_to_errno (ret); }
[tip: sched/core] cpuidle, cpu_pm: Remove RCU fiddling from cpu_pm_{enter,exit}()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 924aed1646bf2859726f7e5dd9265ba14358fbb8 Gitweb: https://git.kernel.org/tip/924aed1646bf2859726f7e5dd9265ba14358fbb8 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:28 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
cpuidle, cpu_pm: Remove RCU fiddling from cpu_pm_{enter,exit}()
All callers should still have RCU enabled.
--- kernel/cpu_pm.c | 9 --------- 1 file changed, 9 deletions(-)
kernel/cpu_pm.c +0 -9
@@ -30,16 +30,9 @@ static int cpu_pm_notify(enum cpu_pm_event event) {int ret; -/* -* This introduces a RCU read critical section, which could be -* disfunctional in cpu idle. Copy RCU_NONIDLE code to let RCU know -* this. -*/ -ct_irq_enter_irqson ();rcu_read_lock (); ret =raw_notifier_call_chain (&cpu_pm_notifier.chain, event,NULL );rcu_read_unlock (); -ct_irq_exit_irqson ();return notifier_to_errno (ret); }
@@ -49,11 +42,9 @@ static int cpu_pm_notify_robust(enum cpu_pm_event event_up, enum cpu_pm_event evunsigned long flags;int ret; -ct_irq_enter_irqson ();raw_spin_lock_irqsave (&cpu_pm_notifier.lock, flags); ret =raw_notifier_call_chain_robust (&cpu_pm_notifier.chain, event_up, event_down,NULL );raw_spin_unlock_irqrestore (&cpu_pm_notifier.lock, flags); -ct_irq_exit_irqson ();return notifier_to_errno (ret); }
Peter Zijlstra
maintainerPATCH v3 15/51 acpi_idle: Remove tracing
In reply to Peter ZijlstraAll the idle routines are called with RCU disabled, as such there must not be any tracing inside.
While there; clean-up the io-port idle thing.
--- drivers/acpi/processor_idle.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/acpi/processor_idle.c +8 -8
@@ -109,8 +109,8 @@ static const struct dmi_system_id proces static void __cpuidle acpi_safe_halt(void) {if (!tif_need_resched ()) { -safe_halt (); -local_irq_disable (); +raw_safe_halt (); +raw_local_irq_disable (); } }
@@ -525,8 +525,11 @@ static int acpi_idle_bm_check(void)return bm_status; } -static void wait_for_freeze (void ) +static __cpuidle void io_idle(unsigned long addr) { +/* IO port based C-state */ +inb (addr); +#ifdef CONFIG_X86 /* No delay is needed if we are in guest */ if (boot_cpu_has (X86_FEATURE_HYPERVISOR ))
@@ -571,9 +574,7 @@ static void __cpuidle acpi_idle_do_entry }else if (cx->entry_method ==ACPI_CSTATE_HALT ) {acpi_safe_halt (); }else { -/* IO port based C-state */ -inb (cx->address); -wait_for_freeze (); +io_idle (cx->address); }perf_lopwr_cb (false);
@@ -595,8 +596,7 @@ static int acpi_idle_play_dead(struct cpif (cx->entry_method ==ACPI_CSTATE_HALT )safe_halt ();else if (cx->entry_method ==ACPI_CSTATE_SYSTEMIO ) { -inb (cx->address); -wait_for_freeze (); +io_idle (cx->address); }else return -ENODEV ;
[tip: sched/core] acpi_idle: Remove tracing
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 8ce78470bf2dd1d08107736e813fa62e09aa4647 Gitweb: https://git.kernel.org/tip/8ce78470bf2dd1d08107736e813fa62e09aa4647 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:29 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
acpi_idle: Remove tracing
All the idle routines are called with RCU disabled, as such there must not be any tracing inside.
While there; clean-up the io-port idle thing.
--- drivers/acpi/processor_idle.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/acpi/processor_idle.c +8 -8
@@ -109,8 +109,8 @@ static const struct dmi_system_id processor_power_dmi_table[] = { static void __cpuidle acpi_safe_halt(void) {if (!tif_need_resched ()) { -safe_halt (); -local_irq_disable (); +raw_safe_halt (); +raw_local_irq_disable (); } }
@@ -523,8 +523,11 @@ static int acpi_idle_bm_check(void)return bm_status; } -static void wait_for_freeze (void ) +static __cpuidle void io_idle(unsigned long addr) { +/* IO port based C-state */ +inb (addr); +#ifdef CONFIG_X86 /* No delay is needed if we are in guest */ if (boot_cpu_has (X86_FEATURE_HYPERVISOR ))
@@ -569,9 +572,7 @@ static void __cpuidle acpi_idle_do_entry(struct acpi_processor_cx *cx) }else if (cx->entry_method ==ACPI_CSTATE_HALT ) {acpi_safe_halt (); }else { -/* IO port based C-state */ -inb (cx->address); -wait_for_freeze (); +io_idle (cx->address); }perf_lopwr_cb (false);
@@ -593,8 +594,7 @@ static int acpi_idle_play_dead(struct cpuidle_device *dev, int index)if (cx->entry_method ==ACPI_CSTATE_HALT )safe_halt ();else if (cx->entry_method ==ACPI_CSTATE_SYSTEMIO ) { -inb (cx->address); -wait_for_freeze (); +io_idle (cx->address); }else return -ENODEV ;
Peter Zijlstra
maintainerPATCH v3 16/51 cpuidle: Annotate poll_idle()
In reply to Peter ZijlstraThe __cpuidle functions will become a noinstr class, as such they need explicit annotations.
--- drivers/cpuidle/poll_state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/cpuidle/poll_state.c +5 -1
@@ -13,7 +13,10 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,struct cpuidle_driver *drv,int index) { -u64 time_start =local_clock (); +u64 time_start; + +instrumentation_begin (); + time_start =local_clock (); dev->poll_time_limit = false;
@@ -39,6 +42,7 @@ static int __cpuidle poll_idle(struct cpraw_local_irq_disable ();current_clr_polling (); +instrumentation_end ();return index; }
[tip: sched/core] cpuidle: Annotate poll_idle()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 00717eb8c9557b5fcdd338db6faef532f6991d63 Gitweb: https://git.kernel.org/tip/00717eb8c9557b5fcdd338db6faef532f6991d63 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:30 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
cpuidle: Annotate poll_idle()
The __cpuidle functions will become a noinstr class, as such they need explicit annotations.
--- drivers/cpuidle/poll_state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/cpuidle/poll_state.c +5 -1
@@ -13,7 +13,10 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,struct cpuidle_driver *drv,int index) { -u64 time_start =local_clock (); +u64 time_start; + +instrumentation_begin (); + time_start =local_clock (); dev->poll_time_limit = false;
@@ -39,6 +42,7 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,raw_local_irq_disable ();current_clr_polling (); +instrumentation_end ();return index; }
Peter Zijlstra
maintainerRe: [PATCH v3 16/51] cpuidle: Annotate poll_idle()
In reply to Peter ZijlstraQuoted 27 lines · Peter Zijlstra
The
__cpuidlefunctions will become a noinstr class, as such they need explicit annotations.Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Rafael J. Wysocki <[email protected]>Tested-by Tony Lindgren <[email protected]>Tested-by Ulf Hansson <[email protected]>--- drivers/cpuidle/poll_state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/cpuidle/poll_state.c +4
-1@@ -13,7 +13,10 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,struct cpuidle_driver *drv,int index) { -u64 time_start =local_clock (); +u64 time_start; + +instrumentation_begin (); + time_start =local_clock (); dev->poll_time_limit = false;
drivers/cpuidle/poll_state.c static int __cpuidle poll_idle(struct cp · line 48
Quoted 7 lines
drivers/cpuidle/poll_state.c +1
-0@@ -39,6 +42,7 @@ static int __cpuidle poll_idle(struct cpraw_local_irq_disable ();current_clr_polling (); +instrumentation_end ();return index; }
Pff, this patch is garbage. However wrote it didn't have his brain engaged :/
Something like the below fixes it, but I still need to build me funny configs like ia64 and paravirt to see if I didn't wreck me something...
arch/x86/kernel/tsc.c +1 -1
@@ -215,7 +215,7 @@ static void __init cyc2ns_init_secondary_cpus(void)/* * Scheduler clock - returns current time in nanosec units. */ -u64 native_sched_clock (void ) +noinstr u64 native_sched_clock(void) {if (static_branch_likely (&__use_tsc)) {u64 tsc_now =rdtsc ();
drivers/cpuidle/cpuidle.c +1 -1
@@ -426,7 +426,7 @@ void cpuidle_reflect(struct cpuidle_device *dev, int index)* @dev: the cpuidle device * */ -u64 cpuidle_poll_time (struct cpuidle_driver *drv, +__cpuidle u64 cpuidle_poll_time(struct cpuidle_driver *drv,struct cpuidle_device *dev) {int i;
drivers/cpuidle/poll_state.c +0 -2
@@ -15,7 +15,6 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, {u64 time_start; -instrumentation_begin (); time_start =local_clock (); dev->poll_time_limit = false;
@@ -42,7 +41,6 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,raw_local_irq_disable ();current_clr_polling (); -instrumentation_end ();return index; }
include/linux/sched/clock.h +2 -2
@@ -45,7 +45,7 @@ static inline u64 cpu_clock(int cpu)return sched_clock (); } -static inline u64 local_clock (void ) +static __always_inline u64 local_clock(void) {return sched_clock (); }
@@ -79,7 +79,7 @@ static inline u64 cpu_clock(int cpu)return sched_clock_cpu (cpu); } -static inline u64 local_clock (void ) +static __always_inline u64 local_clock(void) {return sched_clock_cpu (raw_smp_processor_id ()); }
kernel/sched/clock.c +3 -3
@@ -260,7 +260,7 @@ notrace static inline u64 wrap_max(u64 x, u64 y)* - filter out backward motion * - use the GTOD tick value to create a window to filter crazy TSC values */ -notrace static u64 sched_clock_local(struct sched_clock_data *scd) +noinstr static u64 sched_clock_local(struct sched_clock_data *scd) {u64 now, clock, old_clock, min_clock, max_clock, gtod;s64 delta;
@@ -287,7 +287,7 @@ notrace static u64 sched_clock_local(struct sched_clock_data *scd) clock =wrap_max (clock, min_clock); clock =wrap_min (clock, max_clock); -if (!try_cmpxchg64 (&scd->clock, &old_clock, clock)) +if (!arch_try_cmpxchg64 (&scd->clock, &old_clock, clock)) goto again;return clock;
@@ -360,7 +360,7 @@ notrace static u64 sched_clock_remote(struct sched_clock_data *scd)* * See cpu_clock(). */ -notrace u64 sched_clock_cpu(int cpu) +noinstr u64 sched_clock_cpu(int cpu) {struct sched_clock_data *scd;u64 clock;
Peter Zijlstra
maintainerRe: [PATCH v3 16/51] cpuidle: Annotate poll_idle()
In reply to Peter ZijlstraQuoted 27 lines · Peter Zijlstra
The
__cpuidlefunctions will become a noinstr class, as such they need explicit annotations.Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Rafael J. Wysocki <[email protected]>Tested-by Tony Lindgren <[email protected]>Tested-by Ulf Hansson <[email protected]>--- drivers/cpuidle/poll_state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/cpuidle/poll_state.c +4
-1@@ -13,7 +13,10 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,struct cpuidle_driver *drv,int index) { -u64 time_start =local_clock (); +u64 time_start; + +instrumentation_begin (); + time_start =local_clock (); dev->poll_time_limit = false;
drivers/cpuidle/poll_state.c static int __cpuidle poll_idle(struct cp · line 48
Quoted 7 lines
drivers/cpuidle/poll_state.c +1
-0@@ -39,6 +42,7 @@ static int __cpuidle poll_idle(struct cpraw_local_irq_disable ();current_clr_polling (); +instrumentation_end ();return index; }
Pff, this patch is garbage. However wrote it didn't have his brain engaged :/
Something like the below fixes it, but I still need to build me funny configs like ia64 and paravirt to see if I didn't wreck me something...
arch/x86/kernel/tsc.c +1 -1
@@ -215,7 +215,7 @@ static void __init cyc2ns_init_secondary_cpus(void)/* * Scheduler clock - returns current time in nanosec units. */ -u64 native_sched_clock (void ) +noinstr u64 native_sched_clock(void) {if (static_branch_likely (&__use_tsc)) {u64 tsc_now =rdtsc ();
drivers/cpuidle/cpuidle.c +1 -1
@@ -426,7 +426,7 @@ void cpuidle_reflect(struct cpuidle_device *dev, int index)* @dev: the cpuidle device * */ -u64 cpuidle_poll_time (struct cpuidle_driver *drv, +__cpuidle u64 cpuidle_poll_time(struct cpuidle_driver *drv,struct cpuidle_device *dev) {int i;
drivers/cpuidle/poll_state.c +0 -2
@@ -15,7 +15,6 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, {u64 time_start; -instrumentation_begin (); time_start =local_clock (); dev->poll_time_limit = false;
@@ -42,7 +41,6 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,raw_local_irq_disable ();current_clr_polling (); -instrumentation_end ();return index; }
include/linux/sched/clock.h +2 -2
@@ -45,7 +45,7 @@ static inline u64 cpu_clock(int cpu)return sched_clock (); } -static inline u64 local_clock (void ) +static __always_inline u64 local_clock(void) {return sched_clock (); }
@@ -79,7 +79,7 @@ static inline u64 cpu_clock(int cpu)return sched_clock_cpu (cpu); } -static inline u64 local_clock (void ) +static __always_inline u64 local_clock(void) {return sched_clock_cpu (raw_smp_processor_id ()); }
kernel/sched/clock.c +3 -3
@@ -260,7 +260,7 @@ notrace static inline u64 wrap_max(u64 x, u64 y)* - filter out backward motion * - use the GTOD tick value to create a window to filter crazy TSC values */ -notrace static u64 sched_clock_local(struct sched_clock_data *scd) +noinstr static u64 sched_clock_local(struct sched_clock_data *scd) {u64 now, clock, old_clock, min_clock, max_clock, gtod;s64 delta;
@@ -287,7 +287,7 @@ notrace static u64 sched_clock_local(struct sched_clock_data *scd) clock =wrap_max (clock, min_clock); clock =wrap_min (clock, max_clock); -if (!try_cmpxchg64 (&scd->clock, &old_clock, clock)) +if (!arch_try_cmpxchg64 (&scd->clock, &old_clock, clock)) goto again;return clock;
@@ -360,7 +360,7 @@ notrace static u64 sched_clock_remote(struct sched_clock_data *scd)* * See cpu_clock(). */ -notrace u64 sched_clock_cpu(int cpu) +noinstr u64 sched_clock_cpu(int cpu) {struct sched_clock_data *scd;u64 clock;
Peter Zijlstra
maintainerRe: [PATCH v3 16/51] cpuidle: Annotate poll_idle()
In reply to Peter ZijlstraQuoted 27 lines · Peter Zijlstra
The
__cpuidlefunctions will become a noinstr class, as such they need explicit annotations.Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Rafael J. Wysocki <[email protected]>Tested-by Tony Lindgren <[email protected]>Tested-by Ulf Hansson <[email protected]>--- drivers/cpuidle/poll_state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/cpuidle/poll_state.c +4
-1@@ -13,7 +13,10 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,struct cpuidle_driver *drv,int index) { -u64 time_start =local_clock (); +u64 time_start; + +instrumentation_begin (); + time_start =local_clock (); dev->poll_time_limit = false;
drivers/cpuidle/poll_state.c static int __cpuidle poll_idle(struct cp · line 48
Quoted 7 lines
drivers/cpuidle/poll_state.c +1
-0@@ -39,6 +42,7 @@ static int __cpuidle poll_idle(struct cpraw_local_irq_disable ();current_clr_polling (); +instrumentation_end ();return index; }
Pff, this patch is garbage. However wrote it didn't have his brain engaged :/
Something like the below fixes it, but I still need to build me funny configs like ia64 and paravirt to see if I didn't wreck me something...
arch/x86/kernel/tsc.c +1 -1
@@ -215,7 +215,7 @@ static void __init cyc2ns_init_secondary_cpus(void)/* * Scheduler clock - returns current time in nanosec units. */ -u64 native_sched_clock (void ) +noinstr u64 native_sched_clock(void) {if (static_branch_likely (&__use_tsc)) {u64 tsc_now =rdtsc ();
drivers/cpuidle/cpuidle.c +1 -1
@@ -426,7 +426,7 @@ void cpuidle_reflect(struct cpuidle_device *dev, int index)* @dev: the cpuidle device * */ -u64 cpuidle_poll_time (struct cpuidle_driver *drv, +__cpuidle u64 cpuidle_poll_time(struct cpuidle_driver *drv,struct cpuidle_device *dev) {int i;
drivers/cpuidle/poll_state.c +0 -2
@@ -15,7 +15,6 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, {u64 time_start; -instrumentation_begin (); time_start =local_clock (); dev->poll_time_limit = false;
@@ -42,7 +41,6 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev,raw_local_irq_disable ();current_clr_polling (); -instrumentation_end ();return index; }
include/linux/sched/clock.h +2 -2
@@ -45,7 +45,7 @@ static inline u64 cpu_clock(int cpu)return sched_clock (); } -static inline u64 local_clock (void ) +static __always_inline u64 local_clock(void) {return sched_clock (); }
@@ -79,7 +79,7 @@ static inline u64 cpu_clock(int cpu)return sched_clock_cpu (cpu); } -static inline u64 local_clock (void ) +static __always_inline u64 local_clock(void) {return sched_clock_cpu (raw_smp_processor_id ()); }
kernel/sched/clock.c +3 -3
@@ -260,7 +260,7 @@ notrace static inline u64 wrap_max(u64 x, u64 y)* - filter out backward motion * - use the GTOD tick value to create a window to filter crazy TSC values */ -notrace static u64 sched_clock_local(struct sched_clock_data *scd) +noinstr static u64 sched_clock_local(struct sched_clock_data *scd) {u64 now, clock, old_clock, min_clock, max_clock, gtod;s64 delta;
@@ -287,7 +287,7 @@ notrace static u64 sched_clock_local(struct sched_clock_data *scd) clock =wrap_max (clock, min_clock); clock =wrap_min (clock, max_clock); -if (!try_cmpxchg64 (&scd->clock, &old_clock, clock)) +if (!arch_try_cmpxchg64 (&scd->clock, &old_clock, clock)) goto again;return clock;
@@ -360,7 +360,7 @@ notrace static u64 sched_clock_remote(struct sched_clock_data *scd)* * See cpu_clock(). */ -notrace u64 sched_clock_cpu(int cpu) +noinstr u64 sched_clock_cpu(int cpu) {struct sched_clock_data *scd;u64 clock;
Peter Zijlstra
maintainerPATCH v3 17/51 objtool/idle: Validate __cpuidle code as noinstr
In reply to Peter ZijlstraIdle code is very like entry code in that RCU isn't available. As such, add a little validation.
--- arch/alpha/kernel/vmlinux.lds.S | 1 - arch/arc/kernel/vmlinux.lds.S | 1 - arch/arm/include/asm/vmlinux.lds.h | 1 - arch/arm64/kernel/vmlinux.lds.S | 1 - arch/csky/kernel/vmlinux.lds.S | 1 - arch/hexagon/kernel/vmlinux.lds.S | 1 - arch/ia64/kernel/vmlinux.lds.S | 1 - arch/loongarch/kernel/vmlinux.lds.S | 1 - arch/m68k/kernel/vmlinux-nommu.lds | 1 - arch/m68k/kernel/vmlinux-std.lds | 1 - arch/m68k/kernel/vmlinux-sun3.lds | 1 - arch/microblaze/kernel/vmlinux.lds.S | 1 - arch/mips/kernel/vmlinux.lds.S | 1 - arch/nios2/kernel/vmlinux.lds.S | 1 - arch/openrisc/kernel/vmlinux.lds.S | 1 - arch/parisc/kernel/vmlinux.lds.S | 1 - arch/powerpc/kernel/vmlinux.lds.S | 1 - arch/riscv/kernel/vmlinux-xip.lds.S | 1 - arch/riscv/kernel/vmlinux.lds.S | 1 - arch/s390/kernel/vmlinux.lds.S | 1 - arch/sh/kernel/vmlinux.lds.S | 1 - arch/sparc/kernel/vmlinux.lds.S | 1 - arch/um/kernel/dyn.lds.S | 1 - arch/um/kernel/uml.lds.S | 1 - arch/x86/include/asm/irqflags.h | 11 ++++------- arch/x86/include/asm/mwait.h | 2 +- arch/x86/kernel/vmlinux.lds.S | 1 - arch/xtensa/kernel/vmlinux.lds.S | 1 - include/asm-generic/vmlinux.lds.h | 9 +++------ include/linux/compiler_types.h | 8 ++++++-- include/linux/cpu.h | 3 --- tools/objtool/check.c | 13 +++++++++++++ 32 files changed, 27 insertions(+), 45 deletions(-)
arch/alpha/kernel/vmlinux.lds.S +0 -1
@@ -27,7 +27,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) *(.gnu.warning)
arch/arc/kernel/vmlinux.lds.S +0 -1
@@ -85,7 +85,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/arm/include/asm/vmlinux.lds.h +0 -1
@@ -96,7 +96,6 @@SOFTIRQENTRY_TEXT \ TEXT_TEXT \SCHED_TEXT \ - CPUIDLE_TEXT \ LOCK_TEXT \KPROBES_TEXT \ARM_STUBS_TEXT \
arch/arm64/kernel/vmlinux.lds.S +0 -1
@@ -175,7 +175,6 @@ SECTIONS ENTRY_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT HYPERVISOR_TEXT
arch/csky/kernel/vmlinux.lds.S +0 -1
@@ -34,7 +34,6 @@ SECTIONS SOFTIRQENTRY_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT *(.fixup)
arch/hexagon/kernel/vmlinux.lds.S +0 -1
@@ -41,7 +41,6 @@ SECTIONS IRQENTRY_TEXT SOFTIRQENTRY_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT *(.fixup)
arch/ia64/kernel/vmlinux.lds.S +0 -1
@@ -51,7 +51,6 @@ SECTIONS { __end_ivt_text = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/loongarch/kernel/vmlinux.lds.S +0 -1
@@ -42,7 +42,6 @@ SECTIONS .text : { TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/m68k/kernel/vmlinux-nommu.lds +0 -1
@@ -48,7 +48,6 @@ SECTIONS { IRQENTRY_TEXT SOFTIRQENTRY_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) . = ALIGN(16);
arch/m68k/kernel/vmlinux-std.lds +0 -1
@@ -19,7 +19,6 @@ SECTIONS IRQENTRY_TEXT SOFTIRQENTRY_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) *(.gnu.warning)
arch/m68k/kernel/vmlinux-sun3.lds +0 -1
@@ -19,7 +19,6 @@ SECTIONS IRQENTRY_TEXT SOFTIRQENTRY_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) *(.gnu.warning)
arch/microblaze/kernel/vmlinux.lds.S +0 -1
@@ -36,7 +36,6 @@ SECTIONS { EXIT_TEXT EXIT_CALL SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/mips/kernel/vmlinux.lds.S +0 -1
@@ -61,7 +61,6 @@ SECTIONS .text : { TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/nios2/kernel/vmlinux.lds.S +0 -1
@@ -24,7 +24,6 @@ SECTIONS .text : { TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT
arch/openrisc/kernel/vmlinux.lds.S +0 -1
@@ -52,7 +52,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/parisc/kernel/vmlinux.lds.S +0 -1
@@ -86,7 +86,6 @@ SECTIONS TEXT_TEXT LOCK_TEXT SCHED_TEXT - CPUIDLE_TEXT KPROBES_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT
arch/powerpc/kernel/vmlinux.lds.S +0 -1
@@ -111,7 +111,6 @@ SECTIONS #endif NOINSTR_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/riscv/kernel/vmlinux-xip.lds.S +0 -1
@@ -39,7 +39,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT ENTRY_TEXT
arch/riscv/kernel/vmlinux.lds.S +0 -1
@@ -42,7 +42,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT ENTRY_TEXT
arch/s390/kernel/vmlinux.lds.S +0 -1
@@ -42,7 +42,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/sh/kernel/vmlinux.lds.S +0 -1
@@ -29,7 +29,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/sparc/kernel/vmlinux.lds.S +0 -1
@@ -50,7 +50,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/um/kernel/dyn.lds.S +0 -1
@@ -74,7 +74,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT
arch/um/kernel/uml.lds.S +0 -1
@@ -35,7 +35,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT
arch/x86/include/asm/irqflags.h +4 -7
@@ -8,9 +8,6 @@#include <asm/nospec-branch.h> -/* Provide __cpuidle; we can't safely include <linux/cpu.h> */ -#define __cpuidle __section(".cpuidle.text") -/* * Interrupt control: */
@@ -45,13 +42,13 @@ static __always_inline void native_irq_e asmvolatile ("sti" : : :"memory" ); } -static inline __cpuidle void native_safe_halt(void) +static __always_inline void native_safe_halt(void) {mds_idle_clear_cpu_buffers (); asmvolatile ("sti; hlt" : : :"memory" ); } -static inline __cpuidle void native_halt(void) +static __always_inline void native_halt(void) {mds_idle_clear_cpu_buffers (); asmvolatile ("hlt" : : :"memory" );
@@ -84,7 +81,7 @@ static __always_inline void arch_local_i* Used in the idle loop; sti takes one instruction cycle * to complete: */ -static inline __cpuidle void arch_safe_halt(void) +static __always_inline void arch_safe_halt(void) {native_safe_halt (); }
@@ -93,7 +90,7 @@ static inline __cpuidle void arch_safe_h* Used when interrupts are already enabled or to * shutdown the processor: */ -static inline __cpuidle void halt(void) +static __always_inline void halt(void) {native_halt (); }
arch/x86/include/asm/mwait.h +1 -1
@@ -105,7 +105,7 @@ static inline void __sti_mwait(unsigned* New with Core Duo processors, MWAIT can take some hints based on CPU * capability. */ -static inline void mwait_idle_with_hints (unsigned long eax,unsigned long ecx) +static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned long ecx) {if (static_cpu_has_bug (X86_BUG_MONITOR ) || !current_set_polling_and_test ()) {if (static_cpu_has_bug (X86_BUG_CLFLUSH_MONITOR )) {
arch/x86/kernel/vmlinux.lds.S +0 -1
@@ -129,7 +129,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT SOFTIRQENTRY_TEXT
arch/xtensa/kernel/vmlinux.lds.S +0 -1
@@ -125,7 +125,6 @@ SECTIONS ENTRY_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) }
include/asm-generic/vmlinux.lds.h +3 -6
@@ -580,6 +580,9 @@ALIGN_FUNCTION (); \ __noinstr_text_start = .; \ *(.noinstr.text) \ + __cpuidle_text_start = .; \ + *(.cpuidle.text) \ + __cpuidle_text_end = .; \ __noinstr_text_end = .;/*
@@ -620,12 +623,6 @@ *(.spinlock.text) \ __lock_text_end = .; -#define CPUIDLE_TEXT \ - ALIGN_FUNCTION(); \ - __cpuidle_text_start = .; \ - *(.cpuidle.text) \ - __cpuidle_text_end = .; -#define KPROBES_TEXT \ ALIGN_FUNCTION(); \ __kprobes_text_start = .; \
include/linux/compiler_types.h +6 -2
@@ -232,11 +232,15 @@ struct ftrace_likely_data {#endif /* Section for code which can't be instrumented at all */ -#define noinstr \ - noinline notrace __attribute((__section__(".noinstr.text"))) \ +#define __noinstr_section (section) \ + noinline notrace __attribute((__section__(section))) \ __no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage \ __no_sanitize_memory +#define noinstr __noinstr_section(".noinstr.text") + +#define __cpuidle __noinstr_section(".cpuidle.text") +#endif /* __KERNEL__ */ #endif /* __ASSEMBLY__ */
include/linux/cpu.h +0 -3
@@ -176,9 +176,6 @@ void __noreturn cpu_startup_entry(enum cvoid cpu_idle_poll_ctrl (bool enable); -/* Attach to any functions which should be considered cpuidle. */ -#define __cpuidle __section(".cpuidle.text") -bool cpu_in_idle (unsigned long pc);void arch_cpu_idle (void );
tools/objtool/check.c +13 -0
@@ -376,6 +376,7 @@ static int decode_instructions(struct obif (!strcmp (sec->name,".noinstr.text" ) || !strcmp (sec->name,".entry.text" ) || + !strcmp (sec->name,".cpuidle.text" ) || !strncmp (sec->name,".text.__x86." ,12 )) sec->noinstr = true;
@@ -3361,6 +3362,12 @@ static inline bool noinstr_call_dest(strreturn true;/* +* If the symbol is a static_call trampoline, we can't tell. +*/ +if (func->static_call_tramp) +return true; + +/* * The __ubsan_handle_*() calls are like WARN(), they only happen when * something 'BAD' happened. At the risk of taking the machine down, * let them proceed to get the message out.
@@ -4155,6 +4162,12 @@ static int validate_noinstr_sections(strif (sec) { warnings +=validate_section (file, sec); warnings +=validate_unwind_hints (file, sec); + } + + sec =find_section_by_name (file->elf,".cpuidle.text" ); +if (sec) { + warnings +=validate_section (file, sec); + warnings +=validate_unwind_hints (file, sec); }return warnings;
[tip: sched/core] objtool/idle: Validate __cpuidle code as noinstr
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 2b5a0e425e6e319b1978db1e9564f6af4228a567 Gitweb: https://git.kernel.org/tip/2b5a0e425e6e319b1978db1e9564f6af4228a567 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:31 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
objtool/idle: Validate __cpuidle code as noinstr
Idle code is very like entry code in that RCU isn't available. As such, add a little validation.
--- arch/alpha/kernel/vmlinux.lds.S | 1 - arch/arc/kernel/vmlinux.lds.S | 1 - arch/arm/include/asm/vmlinux.lds.h | 1 - arch/arm64/kernel/vmlinux.lds.S | 1 - arch/csky/kernel/vmlinux.lds.S | 1 - arch/hexagon/kernel/vmlinux.lds.S | 1 - arch/ia64/kernel/vmlinux.lds.S | 1 - arch/loongarch/kernel/vmlinux.lds.S | 1 - arch/m68k/kernel/vmlinux-nommu.lds | 1 - arch/m68k/kernel/vmlinux-std.lds | 1 - arch/m68k/kernel/vmlinux-sun3.lds | 1 - arch/microblaze/kernel/vmlinux.lds.S | 1 - arch/mips/kernel/vmlinux.lds.S | 1 - arch/nios2/kernel/vmlinux.lds.S | 1 - arch/openrisc/kernel/vmlinux.lds.S | 1 - arch/parisc/kernel/vmlinux.lds.S | 1 - arch/powerpc/kernel/vmlinux.lds.S | 1 - arch/riscv/kernel/vmlinux-xip.lds.S | 1 - arch/riscv/kernel/vmlinux.lds.S | 1 - arch/s390/kernel/vmlinux.lds.S | 1 - arch/sh/kernel/vmlinux.lds.S | 1 - arch/sparc/kernel/vmlinux.lds.S | 1 - arch/um/kernel/dyn.lds.S | 1 - arch/um/kernel/uml.lds.S | 1 - arch/x86/include/asm/irqflags.h | 11 ++++------- arch/x86/include/asm/mwait.h | 2 +- arch/x86/kernel/vmlinux.lds.S | 1 - arch/xtensa/kernel/vmlinux.lds.S | 1 - include/asm-generic/vmlinux.lds.h | 9 +++------ include/linux/compiler_types.h | 8 ++++++-- include/linux/cpu.h | 3 --- tools/objtool/check.c | 13 +++++++++++++ 32 files changed, 27 insertions(+), 45 deletions(-)
arch/alpha/kernel/vmlinux.lds.S +0 -1
@@ -27,7 +27,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) *(.gnu.warning)
arch/arc/kernel/vmlinux.lds.S +0 -1
@@ -85,7 +85,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/arm/include/asm/vmlinux.lds.h +0 -1
@@ -96,7 +96,6 @@SOFTIRQENTRY_TEXT \ TEXT_TEXT \SCHED_TEXT \ - CPUIDLE_TEXT \ LOCK_TEXT \KPROBES_TEXT \ARM_STUBS_TEXT \
arch/arm64/kernel/vmlinux.lds.S +0 -1
@@ -175,7 +175,6 @@ SECTIONS ENTRY_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT HYPERVISOR_TEXT
arch/csky/kernel/vmlinux.lds.S +0 -1
@@ -34,7 +34,6 @@ SECTIONS SOFTIRQENTRY_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT *(.fixup)
arch/hexagon/kernel/vmlinux.lds.S +0 -1
@@ -41,7 +41,6 @@ SECTIONS IRQENTRY_TEXT SOFTIRQENTRY_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT *(.fixup)
arch/ia64/kernel/vmlinux.lds.S +0 -1
@@ -51,7 +51,6 @@ SECTIONS { __end_ivt_text = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/loongarch/kernel/vmlinux.lds.S +0 -1
@@ -43,7 +43,6 @@ SECTIONS .text : { TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/m68k/kernel/vmlinux-nommu.lds +0 -1
@@ -48,7 +48,6 @@ SECTIONS { IRQENTRY_TEXT SOFTIRQENTRY_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) . = ALIGN(16);
arch/m68k/kernel/vmlinux-std.lds +0 -1
@@ -19,7 +19,6 @@ SECTIONS IRQENTRY_TEXT SOFTIRQENTRY_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) *(.gnu.warning)
arch/m68k/kernel/vmlinux-sun3.lds +0 -1
@@ -19,7 +19,6 @@ SECTIONS IRQENTRY_TEXT SOFTIRQENTRY_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) *(.gnu.warning)
arch/microblaze/kernel/vmlinux.lds.S +0 -1
@@ -36,7 +36,6 @@ SECTIONS { EXIT_TEXT EXIT_CALL SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/mips/kernel/vmlinux.lds.S +0 -1
@@ -61,7 +61,6 @@ SECTIONS .text : { TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/nios2/kernel/vmlinux.lds.S +0 -1
@@ -24,7 +24,6 @@ SECTIONS .text : { TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT
arch/openrisc/kernel/vmlinux.lds.S +0 -1
@@ -52,7 +52,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/parisc/kernel/vmlinux.lds.S +0 -1
@@ -86,7 +86,6 @@ SECTIONS TEXT_TEXT LOCK_TEXT SCHED_TEXT - CPUIDLE_TEXT KPROBES_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT
arch/powerpc/kernel/vmlinux.lds.S +0 -1
@@ -111,7 +111,6 @@ SECTIONS #endif NOINSTR_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/riscv/kernel/vmlinux-xip.lds.S +0 -1
@@ -39,7 +39,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT ENTRY_TEXT
arch/riscv/kernel/vmlinux.lds.S +0 -1
@@ -42,7 +42,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT ENTRY_TEXT
arch/s390/kernel/vmlinux.lds.S +0 -1
@@ -42,7 +42,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/sh/kernel/vmlinux.lds.S +0 -1
@@ -29,7 +29,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/sparc/kernel/vmlinux.lds.S +0 -1
@@ -50,7 +50,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT IRQENTRY_TEXT
arch/um/kernel/dyn.lds.S +0 -1
@@ -74,7 +74,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT
arch/um/kernel/uml.lds.S +0 -1
@@ -35,7 +35,6 @@ SECTIONS _stext = .; TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT
arch/x86/include/asm/irqflags.h +4 -7
@@ -8,9 +8,6 @@#include <asm/nospec-branch.h> -/* Provide __cpuidle; we can't safely include <linux/cpu.h> */ -#define __cpuidle __section(".cpuidle.text") -/* * Interrupt control: */
@@ -45,13 +42,13 @@ static __always_inline void native_irq_enable(void) asmvolatile ("sti" : : :"memory" ); } -static inline __cpuidle void native_safe_halt(void) +static __always_inline void native_safe_halt(void) {mds_idle_clear_cpu_buffers (); asmvolatile ("sti; hlt" : : :"memory" ); } -static inline __cpuidle void native_halt(void) +static __always_inline void native_halt(void) {mds_idle_clear_cpu_buffers (); asmvolatile ("hlt" : : :"memory" );
@@ -84,7 +81,7 @@ static __always_inline void arch_local_irq_enable(void)* Used in the idle loop; sti takes one instruction cycle * to complete: */ -static inline __cpuidle void arch_safe_halt(void) +static __always_inline void arch_safe_halt(void) {native_safe_halt (); }
@@ -93,7 +90,7 @@ static inline __cpuidle void arch_safe_halt(void)* Used when interrupts are already enabled or to * shutdown the processor: */ -static inline __cpuidle void halt(void) +static __always_inline void halt(void) {native_halt (); }
arch/x86/include/asm/mwait.h +1 -1
@@ -105,7 +105,7 @@ static inline void __sti_mwait(unsigned long eax, unsigned long ecx)* New with Core Duo processors, MWAIT can take some hints based on CPU * capability. */ -static inline void mwait_idle_with_hints (unsigned long eax,unsigned long ecx) +static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned long ecx) {if (static_cpu_has_bug (X86_BUG_MONITOR ) || !current_set_polling_and_test ()) {if (static_cpu_has_bug (X86_BUG_CLFLUSH_MONITOR )) {
arch/x86/kernel/vmlinux.lds.S +0 -1
@@ -129,7 +129,6 @@ SECTIONS HEAD_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT KPROBES_TEXT SOFTIRQENTRY_TEXT
arch/xtensa/kernel/vmlinux.lds.S +0 -1
@@ -125,7 +125,6 @@ SECTIONS ENTRY_TEXT TEXT_TEXT SCHED_TEXT - CPUIDLE_TEXT LOCK_TEXT *(.fixup) }
include/asm-generic/vmlinux.lds.h +3 -6
@@ -558,6 +558,9 @@ALIGN_FUNCTION (); \ __noinstr_text_start = .; \ *(.noinstr.text) \ + __cpuidle_text_start = .; \ + *(.cpuidle.text) \ + __cpuidle_text_end = .; \ __noinstr_text_end = .;/*
@@ -598,12 +601,6 @@ *(.spinlock.text) \ __lock_text_end = .; -#define CPUIDLE_TEXT \ - ALIGN_FUNCTION(); \ - __cpuidle_text_start = .; \ - *(.cpuidle.text) \ - __cpuidle_text_end = .; -#define KPROBES_TEXT \ ALIGN_FUNCTION(); \ __kprobes_text_start = .; \
include/linux/compiler_types.h +6 -2
@@ -232,11 +232,15 @@ struct ftrace_likely_data {#endif /* Section for code which can't be instrumented at all */ -#define noinstr \ - noinline notrace __attribute((__section__(".noinstr.text"))) \ +#define __noinstr_section (section) \ + noinline notrace __attribute((__section__(section))) \ __no_kcsan __no_sanitize_address __no_profile __no_sanitize_coverage \ __no_sanitize_memory +#define noinstr __noinstr_section(".noinstr.text") + +#define __cpuidle __noinstr_section(".cpuidle.text") +#endif /* __KERNEL__ */ #endif /* __ASSEMBLY__ */
include/linux/cpu.h +0 -3
@@ -176,9 +176,6 @@ void __noreturn cpu_startup_entry(enum cpuhp_state state);void cpu_idle_poll_ctrl (bool enable); -/* Attach to any functions which should be considered cpuidle. */ -#define __cpuidle __section(".cpuidle.text") -bool cpu_in_idle (unsigned long pc);void arch_cpu_idle (void );
tools/objtool/check.c +13 -0
@@ -376,6 +376,7 @@ static int decode_instructions(struct objtool_file *file)if (!strcmp (sec->name,".noinstr.text" ) || !strcmp (sec->name,".entry.text" ) || + !strcmp (sec->name,".cpuidle.text" ) || !strncmp (sec->name,".text.__x86." ,12 )) sec->noinstr = true;
@@ -3366,6 +3367,12 @@ static inline bool noinstr_call_dest(struct objtool_file *file,return true;/* +* If the symbol is a static_call trampoline, we can't tell. +*/ +if (func->static_call_tramp) +return true; + +/* * The __ubsan_handle_*() calls are like WARN(), they only happen when * something 'BAD' happened. At the risk of taking the machine down, * let them proceed to get the message out.
@@ -4162,6 +4169,12 @@ static int validate_noinstr_sections(struct objtool_file *file) warnings +=validate_unwind_hints (file, sec); } + sec =find_section_by_name (file->elf,".cpuidle.text" ); +if (sec) { + warnings +=validate_section (file, sec); + warnings +=validate_unwind_hints (file, sec); + } +return warnings; }
Peter Zijlstra
maintainerPATCH v3 18/51 cpuidle,intel_idle: Fix CPUIDLE_FLAG_IRQ_ENABLE *again*
In reply to Peter Zijlstra vmlinux.o: warning: objtool: intel_idle_irq+0x10c: call to trace_hardirqs_off() leaves .noinstr.text section
As per commit 32d4fd5751ea ("cpuidle,intel_idle: Fix CPUIDLE_FLAG_IRQ_ENABLE"):
"must not have tracing in idle functions"
Clearly people can't read and tinker along until splat dissapears. This straight up reverts commit d295ad34f236 ("intel_idle: Fix false positive RCU splats due to incorrect hardirqs state").
It doesn't re-introduce the problem because preceding patches fixed it properly.
--- drivers/idle/intel_idle.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
drivers/idle/intel_idle.c +1 -7
@@ -168,13 +168,7 @@ static __cpuidle int intel_idle_irq(struraw_local_irq_enable (); ret =__intel_idle (dev, drv, index); - -/* -* The lockdep hardirqs state may be changed to 'on' with timer -* tick interrupt followed by __do_softirq(). Use local_irq_disable() -* to keep the hardirqs state correct. -*/ -local_irq_disable (); +raw_local_irq_disable ();return ret; }
[tip: sched/core] cpuidle, intel_idle: Fix CPUIDLE_FLAG_IRQ_ENABLE *again*
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 6d9c7f51b1d9179bf7c3542267c656a934e8af23 Gitweb: https://git.kernel.org/tip/6d9c7f51b1d9179bf7c3542267c656a934e8af23 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:32 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
cpuidle, intel_idle: Fix CPUIDLE_FLAG_IRQ_ENABLE *again*
So objtool found this bug:
vmlinux.o: warning: objtool: intel_idle_irq+0x10c: call to trace_hardirqs_off() leaves .noinstr.text section
As per commit 32d4fd5751ea ("cpuidle,intel_idle: Fix CPUIDLE_FLAG_IRQ_ENABLE"):
"must not have tracing in idle functions"
Clearly people can't read and tinker along until splat dissapears. This straight up reverts commit d295ad34f236 ("intel_idle: Fix false positive RCU splats due to incorrect hardirqs state").
It doesn't re-introduce the problem because preceding patches fixed it properly.
--- drivers/idle/intel_idle.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
drivers/idle/intel_idle.c +1 -7
@@ -168,13 +168,7 @@ static __cpuidle int intel_idle_irq(struct cpuidle_device *dev,raw_local_irq_enable (); ret =__intel_idle (dev, drv, index); - -/* -* The lockdep hardirqs state may be changed to 'on' with timer -* tick interrupt followed by __do_softirq(). Use local_irq_disable() -* to keep the hardirqs state correct. -*/ -local_irq_disable (); +raw_local_irq_disable ();return ret; }
Peter Zijlstra
maintainerPATCH v3 19/51 cpuidle,intel_idle: Fix CPUIDLE_FLAG_INIT_XSTATE
In reply to Peter Zijlstravmlinux.o: warning: objtool: intel_idle_s2idle+0xd5: call to fpu_idle_fpregs() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_xstate+0x11: call to fpu_idle_fpregs() leaves .noinstr.text section vmlinux.o: warning: objtool: fpu_idle_fpregs+0x9: call to xfeatures_in_use() leaves .noinstr.text section
--- arch/x86/include/asm/fpu/xcr.h | 4 ++-- arch/x86/include/asm/special_insns.h | 2 +- arch/x86/kernel/fpu/core.c | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/fpu/xcr.h +2 -2
@@ -5,7 +5,7 @@#define XCR_XFEATURE_ENABLED_MASK 0x00000000#define XCR_XFEATURE_IN_USE_MASK 0x00000001 -static inline u64 xgetbv (u32 index) +static __always_inline u64 xgetbv(u32 index) {u32 eax, edx;
@@ -27,7 +27,7 @@ static inline void xsetbv(u32 index, u64* * Callers should check X86_FEATURE_XGETBV1. */ -static inline u64 xfeatures_in_use (void ) +static __always_inline u64 xfeatures_in_use(void) {return xgetbv (XCR_XFEATURE_IN_USE_MASK ); }
arch/x86/include/asm/special_insns.h +1 -1
@@ -295,7 +295,7 @@ static inline int enqcmds(void __iomem *return 0 ; } -static inline void tile_release (void ) +static __always_inline void tile_release(void) {/* * Instruction opcode for TILERELEASE; supported in binutils
arch/x86/kernel/fpu/core.c +2 -2
@@ -856,12 +856,12 @@ int fpu__exception_code(struct fpu *fpu,* Initialize register state that may prevent from entering low-power idle. * This function will be invoked from the cpuidle driver only when needed. */ -void fpu_idle_fpregs (void ) +noinstr void fpu_idle_fpregs(void) {/* Note: AMX_TILE being enabled implies XGETBV1 support */ if (cpu_feature_enabled (X86_FEATURE_AMX_TILE ) && (xfeatures_in_use () &XFEATURE_MASK_XTILE )) {tile_release (); -fpregs_deactivate (¤t->thread.fpu); +__this_cpu_write (fpu_fpregs_owner_ctx,NULL ); } }
[tip: sched/core] cpuidle, intel_idle: Fix CPUIDLE_FLAG_INIT_XSTATE
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 821ad23d0eaff73ef599ece39ecc77482df20a8c Gitweb: https://git.kernel.org/tip/821ad23d0eaff73ef599ece39ecc77482df20a8c Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:33 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
cpuidle, intel_idle: Fix CPUIDLE_FLAG_INIT_XSTATE
Fix instrumentation bugs objtool found:
vmlinux.o: warning: objtool: intel_idle_s2idle+0xd5: call to fpu_idle_fpregs() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_xstate+0x11: call to fpu_idle_fpregs() leaves .noinstr.text section vmlinux.o: warning: objtool: fpu_idle_fpregs+0x9: call to xfeatures_in_use() leaves .noinstr.text section
--- arch/x86/include/asm/fpu/xcr.h | 4 ++-- arch/x86/include/asm/special_insns.h | 2 +- arch/x86/kernel/fpu/core.c | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/fpu/xcr.h +2 -2
@@ -5,7 +5,7 @@#define XCR_XFEATURE_ENABLED_MASK 0x00000000#define XCR_XFEATURE_IN_USE_MASK 0x00000001 -static inline u64 xgetbv (u32 index) +static __always_inline u64 xgetbv(u32 index) {u32 eax, edx;
@@ -27,7 +27,7 @@ static inline void xsetbv(u32 index, u64 value)* * Callers should check X86_FEATURE_XGETBV1. */ -static inline u64 xfeatures_in_use (void ) +static __always_inline u64 xfeatures_in_use(void) {return xgetbv (XCR_XFEATURE_IN_USE_MASK ); }
arch/x86/include/asm/special_insns.h +1 -1
@@ -295,7 +295,7 @@ static inline int enqcmds(void __iomem *dst, const void *src)return 0 ; } -static inline void tile_release (void ) +static __always_inline void tile_release(void) {/* * Instruction opcode for TILERELEASE; supported in binutils
arch/x86/kernel/fpu/core.c +2 -2
@@ -853,12 +853,12 @@ int fpu__exception_code(struct fpu *fpu, int trap_nr)* Initialize register state that may prevent from entering low-power idle. * This function will be invoked from the cpuidle driver only when needed. */ -void fpu_idle_fpregs (void ) +noinstr void fpu_idle_fpregs(void) {/* Note: AMX_TILE being enabled implies XGETBV1 support */ if (cpu_feature_enabled (X86_FEATURE_AMX_TILE ) && (xfeatures_in_use () &XFEATURE_MASK_XTILE )) {tile_release (); -fpregs_deactivate (¤t->thread.fpu); +__this_cpu_write (fpu_fpregs_owner_ctx,NULL ); } }
Peter Zijlstra
maintainerPATCH v3 20/51 cpuidle,intel_idle: Fix CPUIDLE_FLAG_IBRS
In reply to Peter Zijlstravmlinux.o: warning: objtool: intel_idle_ibrs+0x17: call to spec_ctrl_current() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_ibrs+0x27: call to wrmsrl.constprop.0() leaves .noinstr.text section
--- arch/x86/kernel/cpu/bugs.c | 2 +- drivers/idle/intel_idle.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/bugs.c +1 -1
@@ -79,7 +79,7 @@ void write_spec_ctrl_current(u64 val, bowrmsrl (MSR_IA32_SPEC_CTRL , val); } -u64 spec_ctrl_current (void ) +noinstr u64 spec_ctrl_current(void) {return this_cpu_read (x86_spec_ctrl_current); }
drivers/idle/intel_idle.c +2 -2
@@ -181,12 +181,12 @@ static __cpuidle int intel_idle_ibrs(strint ret;if (smt_active) -wrmsrl (MSR_IA32_SPEC_CTRL ,0 ); +native_wrmsrl (MSR_IA32_SPEC_CTRL ,0 ); ret =__intel_idle (dev, drv, index);if (smt_active) -wrmsrl (MSR_IA32_SPEC_CTRL , spec_ctrl); +native_wrmsrl (MSR_IA32_SPEC_CTRL , spec_ctrl);return ret; }
[tip: sched/core] cpuidle, intel_idle: Fix CPUIDLE_FLAG_IBRS
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 9b461a6faae7b220c32466261965778b10189e54 Gitweb: https://git.kernel.org/tip/9b461a6faae7b220c32466261965778b10189e54 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:34 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
cpuidle, intel_idle: Fix CPUIDLE_FLAG_IBRS
objtool to the rescue:
vmlinux.o: warning: objtool: intel_idle_ibrs+0x17: call to spec_ctrl_current() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_ibrs+0x27: call to wrmsrl.constprop.0() leaves .noinstr.text section
--- arch/x86/kernel/cpu/bugs.c | 2 +- drivers/idle/intel_idle.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/bugs.c +1 -1
@@ -86,7 +86,7 @@ void update_spec_ctrl_cond(u64 val)wrmsrl (MSR_IA32_SPEC_CTRL , val); } -u64 spec_ctrl_current (void ) +noinstr u64 spec_ctrl_current(void) {return this_cpu_read (x86_spec_ctrl_current); }
drivers/idle/intel_idle.c +2 -2
@@ -181,12 +181,12 @@ static __cpuidle int intel_idle_ibrs(struct cpuidle_device *dev,int ret;if (smt_active) -wrmsrl (MSR_IA32_SPEC_CTRL ,0 ); +native_wrmsrl (MSR_IA32_SPEC_CTRL ,0 ); ret =__intel_idle (dev, drv, index);if (smt_active) -wrmsrl (MSR_IA32_SPEC_CTRL , spec_ctrl); +native_wrmsrl (MSR_IA32_SPEC_CTRL , spec_ctrl);return ret; }
Peter Zijlstra
maintainerPATCH v3 21/51 arch/idle: Change arch_cpu_idle() IRQ behaviour
In reply to Peter ZijlstraCurrent arch_cpu_idle() is called with IRQs disabled, but will return with IRQs enabled.
However, the very first thing the generic code does after calling arch_cpu_idle() is raw_local_irq_disable(). This means that architectures that can idle with IRQs disabled end up doing a pointless 'enable-disable' dance.
Therefore, push this IRQ disabling into the idle function, meaning that those architectures can avoid the pointless IRQ state flipping.
--- arch/alpha/kernel/process.c | 1 - arch/arc/kernel/process.c | 3 +++ arch/arm/kernel/process.c | 1 - arch/arm/mach-gemini/board-dt.c | 3 ++- arch/arm64/kernel/idle.c | 1 - arch/csky/kernel/process.c | 1 - arch/csky/kernel/smp.c | 2 +- arch/hexagon/kernel/process.c | 1 - arch/ia64/kernel/process.c | 1 + arch/loongarch/kernel/idle.c | 1 + arch/microblaze/kernel/process.c | 1 - arch/mips/kernel/idle.c | 8 +++----- arch/nios2/kernel/process.c | 1 - arch/openrisc/kernel/process.c | 1 + arch/parisc/kernel/process.c | 2 -- arch/powerpc/kernel/idle.c | 5 ++--- arch/riscv/kernel/process.c | 1 - arch/s390/kernel/idle.c | 1 - arch/sh/kernel/idle.c | 1 + arch/sparc/kernel/leon_pmc.c | 4 ++++ arch/sparc/kernel/process_32.c | 1 - arch/sparc/kernel/process_64.c | 3 ++- arch/um/kernel/process.c | 1 - arch/x86/coco/tdx/tdx.c | 3 +++ arch/x86/kernel/process.c | 15 ++++----------- arch/xtensa/kernel/process.c | 1 + kernel/sched/idle.c | 2 -- 27 files changed, 29 insertions(+), 37 deletions(-)
arch/alpha/kernel/process.c +0 -1
@@ -57,7 +57,6 @@ EXPORT_SYMBOL(pm_power_off);void arch_cpu_idle (void ) {wtint (0 ); -raw_local_irq_enable (); }void arch_cpu_idle_dead (void )
arch/arc/kernel/process.c +3 -0
@@ -114,6 +114,8 @@ void arch_cpu_idle(void) "sleep %0 \n" : :"I"(arg)); /* can't be "r" has to be embedded const */ + + raw_local_irq_disable(); }#else /* ARC700 */
@@ -122,6 +124,7 @@ void arch_cpu_idle(void) {/* sleep, but enable both set E1/E2 (levels of interrupts) before committing */ __asm__ __volatile__("sleep 0x3 \n" ); +raw_local_irq_disable (); }#endif
arch/arm/kernel/process.c +0 -1
@@ -78,7 +78,6 @@ void arch_cpu_idle(void)arm_pm_idle (); elsecpu_do_idle (); -raw_local_irq_enable (); }void arch_cpu_idle_prepare (void )
arch/arm/mach-gemini/board-dt.c +2 -1
@@ -42,8 +42,9 @@ static void gemini_idle(void)*/ /* FIXME: Enabling interrupts here is racy! */ -local_irq_enable (); +raw_local_irq_enable ();cpu_do_idle (); +raw_local_irq_disable (); }static void __init gemini_init_machine (void )
arch/arm64/kernel/idle.c +0 -1
@@ -42,5 +42,4 @@ void noinstr arch_cpu_idle(void)* tricks */ cpu_do_idle (); -raw_local_irq_enable (); }
arch/csky/kernel/process.c +0 -1
@@ -100,6 +100,5 @@ void arch_cpu_idle(void)#ifdef CONFIG_CPU_PM_STOP asmvolatile ("stop\n" ); #endif - raw_local_irq_enable(); }#endif
arch/csky/kernel/smp.c +1 -1
@@ -309,7 +309,7 @@ void arch_cpu_idle_dead(void)while (!secondary_stack)arch_cpu_idle (); -local_irq_disable (); +raw_local_irq_disable (); asmvolatile ( "mov sp, %0\n"
arch/hexagon/kernel/process.c +0 -1
@@ -44,7 +44,6 @@ void arch_cpu_idle(void) {__vmwait ();/* interrupts wake us up, but irqs are still disabled */ -raw_local_irq_enable (); }/*
arch/ia64/kernel/process.c +1 -0
@@ -242,6 +242,7 @@ void arch_cpu_idle(void) (*mark_idle)(1 );raw_safe_halt (); +raw_local_irq_disable ();if (mark_idle) (*mark_idle)(0 );
arch/loongarch/kernel/idle.c +1 -0
@@ -13,4 +13,5 @@ void __cpuidle arch_cpu_idle(void) {raw_local_irq_enable ();__arch_cpu_idle ();/* idle instruction needs irq enabled */ +raw_local_irq_disable (); }
arch/microblaze/kernel/process.c +0 -1
@@ -140,5 +140,4 @@ int dump_fpu(struct pt_regs *regs, elf_fvoid arch_cpu_idle (void ) { -raw_local_irq_enable (); }
arch/mips/kernel/idle.c +3 -5
@@ -33,13 +33,13 @@ static void __cpuidle r3081_wait(void) {unsigned long cfg =read_c0_conf ();write_c0_conf (cfg |R30XX_CONF_HALT ); -raw_local_irq_enable (); } void __cpuidle r4k_wait(void) {raw_local_irq_enable ();__r4k_wait (); +raw_local_irq_disable (); }/*
@@ -57,7 +57,6 @@ void __cpuidle r4k_wait_irqoff(void)" .set arch=r4000 \n" " wait \n" " .set pop \n"); -raw_local_irq_enable (); }/*
@@ -77,7 +76,6 @@ static void __cpuidle rm7k_wait_irqoff(v" wait \n" " mtc0 $1, $12 # stalls until W stage \n" " .set pop \n"); -raw_local_irq_enable (); }/*
@@ -103,6 +101,8 @@ static void __cpuidle au1k_wait(void) " nop \n" " .set pop \n" : : "r" (au1k_wait), "r" (c0status)); + +raw_local_irq_disable (); }static int __initdata nowait;
@@ -245,8 +245,6 @@ void arch_cpu_idle(void) {if (cpu_wait)cpu_wait (); -else -raw_local_irq_enable (); }#ifdef CONFIG_CPU_IDLE
arch/nios2/kernel/process.c +0 -1
@@ -33,7 +33,6 @@ EXPORT_SYMBOL(pm_power_off);void arch_cpu_idle (void ) { -raw_local_irq_enable (); }/*
arch/openrisc/kernel/process.c +1 -0
@@ -102,6 +102,7 @@ void arch_cpu_idle(void)raw_local_irq_enable ();if (mfspr (SPR_UPR ) &SPR_UPR_PMP )mtspr (SPR_PMR ,mfspr (SPR_PMR ) |SPR_PMR_DME ); +raw_local_irq_disable (); }void (*pm_power_off)(void ) =NULL ;
arch/parisc/kernel/process.c +0 -2
@@ -187,8 +187,6 @@ void arch_cpu_idle_dead(void) void __cpuidle arch_cpu_idle(void) { -raw_local_irq_enable (); -/* nop on real hardware, qemu will idle sleep. */ asmvolatile ("or %%r10,%%r10,%%r10\n" :::); }
arch/powerpc/kernel/idle.c +2 -3
@@ -51,10 +51,9 @@ void arch_cpu_idle(void)* Some power_save functions return with * interrupts enabled, some don't. */ -if (irqs_disabled ()) -raw_local_irq_enable (); +if (!irqs_disabled ()) +raw_local_irq_disable (); }else { -raw_local_irq_enable ();/* * Go into low thread priority and possibly * low power mode.
arch/riscv/kernel/process.c +0 -1
@@ -39,7 +39,6 @@ extern asmlinkage void ret_from_kernel_tvoid arch_cpu_idle (void ) {cpu_do_idle (); -raw_local_irq_enable (); }void __show_regs (struct pt_regs *regs)
arch/s390/kernel/idle.c +0 -1
@@ -66,7 +66,6 @@ void arch_cpu_idle(void) idle->idle_count++;account_idle_time (cputime_to_nsecs (idle_time));raw_write_seqcount_end (&idle->seqcount); -raw_local_irq_enable (); } static ssize_t show_idle_count(struct device *dev,
arch/sh/kernel/idle.c +1 -0
@@ -25,6 +25,7 @@ void default_idle(void)raw_local_irq_enable ();/* Isn't this racy ? */ cpu_sleep (); +raw_local_irq_disable ();clear_bl_bit (); }
arch/sparc/kernel/leon_pmc.c +4 -0
@@ -57,6 +57,8 @@ static void pmc_leon_idle_fixup(void) "lda [%0] %1, %%g0\n" : : "r"(address), "i"(ASI_LEON_BYPASS)); + +raw_local_irq_disable (); }/*
@@ -70,6 +72,8 @@ static void pmc_leon_idle(void)/* For systems without power-down, this will be no-op */ __asm__ __volatile__ ("wr %g0, %asr19\n\t" ); + +raw_local_irq_disable (); }/* Install LEON Power Down function */
arch/sparc/kernel/process_32.c +0 -1
@@ -71,7 +71,6 @@ void arch_cpu_idle(void) {if (sparc_idle) (*sparc_idle)(); -raw_local_irq_enable (); }/* XXX cli/sti -> local_irq_xxx here, check this works once SMP is fixed. */
arch/sparc/kernel/process_64.c +2 -1
@@ -59,7 +59,6 @@ void arch_cpu_idle(void) {if (tlb_type != hypervisor) {touch_nmi_watchdog (); -raw_local_irq_enable (); }else {unsigned long pstate;
@@ -90,6 +89,8 @@ void arch_cpu_idle(void) "wrpr %0, %%g0, %%pstate" : "=&r" (pstate) : "i" (PSTATE_IE)); + +raw_local_irq_disable (); } }
arch/um/kernel/process.c +0 -1
@@ -217,7 +217,6 @@ void arch_cpu_idle(void) { cpu_tasks[current_thread_info ()->cpu].pid =os_getpid ();um_idle_sleep (); -raw_local_irq_enable (); }int __cant_sleep (void ) {
arch/x86/coco/tdx/tdx.c +3 -0
@@ -223,6 +223,9 @@ void __cpuidle tdx_safe_halt(void)*/ if (__halt (irq_disabled, do_sti))WARN_ONCE (1 ,"HLT instruction emulation failed\n" ); + +/* XXX I can't make sense of what @do_sti actually does */ +raw_local_irq_disable (); }static int read_msr (struct pt_regs *regs,struct ve_info *ve)
arch/x86/kernel/process.c +4 -11
@@ -701,6 +701,7 @@ EXPORT_SYMBOL(boot_option_idle_override) void __cpuidle default_idle(void) {raw_safe_halt (); +raw_local_irq_disable (); }#if defined(CONFIG_APM_MODULE ) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE )EXPORT_SYMBOL (default_idle);
@@ -806,13 +807,7 @@ static void amd_e400_idle(void)default_idle (); -/* -* The switch back from broadcast mode needs to be called with -* interrupts disabled. -*/ -raw_local_irq_disable ();tick_broadcast_exit (); -raw_local_irq_enable (); }/*
@@ -870,12 +865,10 @@ static __cpuidle void mwait_idle(void) }__monitor ((void *)¤t_thread_info ()->flags,0 ,0 ); -if (!need_resched ()) +if (!need_resched ()) {__sti_mwait (0 ,0 ); -else -raw_local_irq_enable (); - }else { -raw_local_irq_enable (); +raw_local_irq_disable (); + } }__current_clr_polling (); }
arch/xtensa/kernel/process.c +1 -0
@@ -183,6 +183,7 @@ void coprocessor_flush_release_all(strucvoid arch_cpu_idle (void ) {platform_idle (); +raw_local_irq_disable (); }/*
kernel/sched/idle.c +0 -2
@@ -79,7 +79,6 @@ void __weak arch_cpu_idle_dead(void) { }void __weak arch_cpu_idle (void ) { cpu_idle_force_poll =1 ; -raw_local_irq_enable (); }/**
@@ -96,7 +95,6 @@ void __cpuidle default_idle_call(void)ct_cpuidle_enter ();arch_cpu_idle (); -raw_local_irq_disable ();ct_cpuidle_exit ();start_critical_timings ();
[tip: sched/core] arch/idle: Change arch_cpu_idle() behavior: always exit with IRQs disabled
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 89b3098703bd2aa3237ef10a704e6a5838e6ea69 Gitweb: https://git.kernel.org/tip/89b3098703bd2aa3237ef10a704e6a5838e6ea69 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:35 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
arch/idle: Change arch_cpu_idle() behavior: always exit with IRQs disabled
Current arch_cpu_idle() is called with IRQs disabled, but will return with IRQs enabled.
However, the very first thing the generic code does after calling arch_cpu_idle() is raw_local_irq_disable(). This means that architectures that can idle with IRQs disabled end up doing a pointless 'enable-disable' dance.
Therefore, push this IRQ disabling into the idle function, meaning that those architectures can avoid the pointless IRQ state flipping.
--- arch/alpha/kernel/process.c | 1 - arch/arc/kernel/process.c | 3 +++ arch/arm/kernel/process.c | 1 - arch/arm/mach-gemini/board-dt.c | 3 ++- arch/arm64/kernel/idle.c | 1 - arch/csky/kernel/process.c | 1 - arch/csky/kernel/smp.c | 2 +- arch/hexagon/kernel/process.c | 1 - arch/ia64/kernel/process.c | 1 + arch/loongarch/kernel/idle.c | 1 + arch/microblaze/kernel/process.c | 1 - arch/mips/kernel/idle.c | 8 +++----- arch/nios2/kernel/process.c | 1 - arch/openrisc/kernel/process.c | 1 + arch/parisc/kernel/process.c | 2 -- arch/powerpc/kernel/idle.c | 5 ++--- arch/riscv/kernel/process.c | 1 - arch/s390/kernel/idle.c | 1 - arch/sh/kernel/idle.c | 1 + arch/sparc/kernel/leon_pmc.c | 4 ++++ arch/sparc/kernel/process_32.c | 1 - arch/sparc/kernel/process_64.c | 3 ++- arch/um/kernel/process.c | 1 - arch/x86/coco/tdx/tdx.c | 3 +++ arch/x86/kernel/process.c | 15 ++++----------- arch/xtensa/kernel/process.c | 1 + kernel/sched/idle.c | 2 -- 27 files changed, 29 insertions(+), 37 deletions(-)
arch/alpha/kernel/process.c +0 -1
@@ -57,7 +57,6 @@ EXPORT_SYMBOL(pm_power_off);void arch_cpu_idle (void ) {wtint (0 ); -raw_local_irq_enable (); }void arch_cpu_idle_dead (void )
arch/arc/kernel/process.c +3 -0
@@ -114,6 +114,8 @@ void arch_cpu_idle(void) "sleep %0 \n" : :"I"(arg)); /* can't be "r" has to be embedded const */ + + raw_local_irq_disable(); }#else /* ARC700 */
@@ -122,6 +124,7 @@ void arch_cpu_idle(void) {/* sleep, but enable both set E1/E2 (levels of interrupts) before committing */ __asm__ __volatile__("sleep 0x3 \n" ); +raw_local_irq_disable (); }#endif
arch/arm/kernel/process.c +0 -1
@@ -78,7 +78,6 @@ void arch_cpu_idle(void)arm_pm_idle (); elsecpu_do_idle (); -raw_local_irq_enable (); }void arch_cpu_idle_prepare (void )
arch/arm/mach-gemini/board-dt.c +2 -1
@@ -42,8 +42,9 @@ static void gemini_idle(void)*/ /* FIXME: Enabling interrupts here is racy! */ -local_irq_enable (); +raw_local_irq_enable ();cpu_do_idle (); +raw_local_irq_disable (); }static void __init gemini_init_machine (void )
arch/arm64/kernel/idle.c +0 -1
@@ -42,5 +42,4 @@ void noinstr arch_cpu_idle(void)* tricks */ cpu_do_idle (); -raw_local_irq_enable (); }
arch/csky/kernel/process.c +0 -1
@@ -100,6 +100,5 @@ void arch_cpu_idle(void)#ifdef CONFIG_CPU_PM_STOP asmvolatile ("stop\n" ); #endif - raw_local_irq_enable(); }#endif
arch/csky/kernel/smp.c +1 -1
@@ -309,7 +309,7 @@ void arch_cpu_idle_dead(void)while (!secondary_stack)arch_cpu_idle (); -local_irq_disable (); +raw_local_irq_disable (); asmvolatile ( "mov sp, %0\n"
arch/hexagon/kernel/process.c +0 -1
@@ -44,7 +44,6 @@ void arch_cpu_idle(void) {__vmwait ();/* interrupts wake us up, but irqs are still disabled */ -raw_local_irq_enable (); }/*
arch/ia64/kernel/process.c +1 -0
@@ -242,6 +242,7 @@ void arch_cpu_idle(void) (*mark_idle)(1 );raw_safe_halt (); +raw_local_irq_disable ();if (mark_idle) (*mark_idle)(0 );
arch/loongarch/kernel/idle.c +1 -0
@@ -13,4 +13,5 @@ void __cpuidle arch_cpu_idle(void) {raw_local_irq_enable ();__arch_cpu_idle ();/* idle instruction needs irq enabled */ +raw_local_irq_disable (); }
arch/microblaze/kernel/process.c +0 -1
@@ -140,5 +140,4 @@ int elf_core_copy_task_fpregs(struct task_struct *t, elf_fpregset_t *fpu)void arch_cpu_idle (void ) { -raw_local_irq_enable (); }
arch/mips/kernel/idle.c +3 -5
@@ -33,13 +33,13 @@ static void __cpuidle r3081_wait(void) {unsigned long cfg =read_c0_conf ();write_c0_conf (cfg |R30XX_CONF_HALT ); -raw_local_irq_enable (); } void __cpuidle r4k_wait(void) {raw_local_irq_enable ();__r4k_wait (); +raw_local_irq_disable (); }/*
@@ -57,7 +57,6 @@ void __cpuidle r4k_wait_irqoff(void)" .set arch=r4000 \n" " wait \n" " .set pop \n"); -raw_local_irq_enable (); }/*
@@ -77,7 +76,6 @@ static void __cpuidle rm7k_wait_irqoff(void)" wait \n" " mtc0 $1, $12 # stalls until W stage \n" " .set pop \n"); -raw_local_irq_enable (); }/*
@@ -103,6 +101,8 @@ static void __cpuidle au1k_wait(void) " nop \n" " .set pop \n" : : "r" (au1k_wait), "r" (c0status)); + +raw_local_irq_disable (); }static int __initdata nowait;
@@ -245,8 +245,6 @@ void arch_cpu_idle(void) {if (cpu_wait)cpu_wait (); -else -raw_local_irq_enable (); }#ifdef CONFIG_CPU_IDLE
arch/nios2/kernel/process.c +0 -1
@@ -33,7 +33,6 @@ EXPORT_SYMBOL(pm_power_off);void arch_cpu_idle (void ) { -raw_local_irq_enable (); }/*
arch/openrisc/kernel/process.c +1 -0
@@ -102,6 +102,7 @@ void arch_cpu_idle(void)raw_local_irq_enable ();if (mfspr (SPR_UPR ) &SPR_UPR_PMP )mtspr (SPR_PMR ,mfspr (SPR_PMR ) |SPR_PMR_DME ); +raw_local_irq_disable (); }void (*pm_power_off)(void ) =NULL ;
arch/parisc/kernel/process.c +0 -2
@@ -183,8 +183,6 @@ void arch_cpu_idle_dead(void) void __cpuidle arch_cpu_idle(void) { -raw_local_irq_enable (); -/* nop on real hardware, qemu will idle sleep. */ asmvolatile ("or %%r10,%%r10,%%r10\n" :::); }
arch/powerpc/kernel/idle.c +2 -3
@@ -51,10 +51,9 @@ void arch_cpu_idle(void)* Some power_save functions return with * interrupts enabled, some don't. */ -if (irqs_disabled ()) -raw_local_irq_enable (); +if (!irqs_disabled ()) +raw_local_irq_disable (); }else { -raw_local_irq_enable ();/* * Go into low thread priority and possibly * low power mode.
arch/riscv/kernel/process.c +0 -1
@@ -39,7 +39,6 @@ extern asmlinkage void ret_from_kernel_thread(void);void arch_cpu_idle (void ) {cpu_do_idle (); -raw_local_irq_enable (); }void __show_regs (struct pt_regs *regs)
arch/s390/kernel/idle.c +0 -1
@@ -66,7 +66,6 @@ void arch_cpu_idle(void) idle->idle_count++;account_idle_time (cputime_to_nsecs (idle_time));raw_write_seqcount_end (&idle->seqcount); -raw_local_irq_enable (); } static ssize_t show_idle_count(struct device *dev,
arch/sh/kernel/idle.c +1 -0
@@ -25,6 +25,7 @@ void default_idle(void)raw_local_irq_enable ();/* Isn't this racy ? */ cpu_sleep (); +raw_local_irq_disable ();clear_bl_bit (); }
arch/sparc/kernel/leon_pmc.c +4 -0
@@ -57,6 +57,8 @@ static void pmc_leon_idle_fixup(void) "lda [%0] %1, %%g0\n" : : "r"(address), "i"(ASI_LEON_BYPASS)); + +raw_local_irq_disable (); }/*
@@ -70,6 +72,8 @@ static void pmc_leon_idle(void)/* For systems without power-down, this will be no-op */ __asm__ __volatile__ ("wr %g0, %asr19\n\t" ); + +raw_local_irq_disable (); }/* Install LEON Power Down function */
arch/sparc/kernel/process_32.c +0 -1
@@ -71,7 +71,6 @@ void arch_cpu_idle(void) {if (sparc_idle) (*sparc_idle)(); -raw_local_irq_enable (); }/* XXX cli/sti -> local_irq_xxx here, check this works once SMP is fixed. */
arch/sparc/kernel/process_64.c +2 -1
@@ -59,7 +59,6 @@ void arch_cpu_idle(void) {if (tlb_type != hypervisor) {touch_nmi_watchdog (); -raw_local_irq_enable (); }else {unsigned long pstate;
@@ -90,6 +89,8 @@ void arch_cpu_idle(void) "wrpr %0, %%g0, %%pstate" : "=&r" (pstate) : "i" (PSTATE_IE)); + +raw_local_irq_disable (); } }
arch/um/kernel/process.c +0 -1
@@ -218,7 +218,6 @@ void arch_cpu_idle(void) { cpu_tasks[current_thread_info ()->cpu].pid =os_getpid ();um_idle_sleep (); -raw_local_irq_enable (); }int __cant_sleep (void ) {
arch/x86/coco/tdx/tdx.c +3 -0
@@ -274,6 +274,9 @@ void __cpuidle tdx_safe_halt(void)*/ if (__halt (irq_disabled, do_sti))WARN_ONCE (1 ,"HLT instruction emulation failed\n" ); + +/* XXX I can't make sense of what @do_sti actually does */ +raw_local_irq_disable (); }static int read_msr (struct pt_regs *regs,struct ve_info *ve)
arch/x86/kernel/process.c +4 -11
@@ -701,6 +701,7 @@ EXPORT_SYMBOL(boot_option_idle_override); void __cpuidle default_idle(void) {raw_safe_halt (); +raw_local_irq_disable (); }#if defined(CONFIG_APM_MODULE ) || defined(CONFIG_HALTPOLL_CPUIDLE_MODULE )EXPORT_SYMBOL (default_idle);
@@ -806,13 +807,7 @@ static void amd_e400_idle(void)default_idle (); -/* -* The switch back from broadcast mode needs to be called with -* interrupts disabled. -*/ -raw_local_irq_disable ();tick_broadcast_exit (); -raw_local_irq_enable (); }/*
@@ -870,12 +865,10 @@ static __cpuidle void mwait_idle(void) }__monitor ((void *)¤t_thread_info ()->flags,0 ,0 ); -if (!need_resched ()) +if (!need_resched ()) {__sti_mwait (0 ,0 ); -else -raw_local_irq_enable (); - }else { -raw_local_irq_enable (); +raw_local_irq_disable (); + } }__current_clr_polling (); }
arch/xtensa/kernel/process.c +1 -0
@@ -183,6 +183,7 @@ void coprocessor_flush_release_all(struct thread_info *ti)void arch_cpu_idle (void ) {platform_idle (); +raw_local_irq_disable (); }/*
kernel/sched/idle.c +0 -2
@@ -79,7 +79,6 @@ void __weak arch_cpu_idle_dead(void) { }void __weak arch_cpu_idle (void ) { cpu_idle_force_poll =1 ; -raw_local_irq_enable (); }/**
@@ -96,7 +95,6 @@ void __cpuidle default_idle_call(void)ct_cpuidle_enter ();arch_cpu_idle (); -raw_local_irq_disable ();ct_cpuidle_exit ();start_critical_timings ();
Peter Zijlstra
maintainerPATCH v3 22/51 x86/tdx: Remove TDX_HCALL_ISSUE_STI
In reply to Peter ZijlstraNow that arch_cpu_idle() is expected to return with IRQs disabled, avoid the useless STI/CLI dance.
Per the specs this is supposed to work, but nobody has yet relied up this behaviour so broken implementations are possible.
--- arch/x86/coco/tdx/tdcall.S | 13 ------------- arch/x86/coco/tdx/tdx.c | 23 ++++------------------- arch/x86/include/asm/shared/tdx.h | 1 - 3 files changed, 4 insertions(+), 33 deletions(-)
arch/x86/coco/tdx/tdcall.S +0 -13
@@ -139,19 +139,6 @@ SYM_FUNC_START(__tdx_hypercall) movl $TDVMCALL_EXPOSE_REGS_MASK, %ecx - /* - * For the idle loop STI needs to be called directly before the TDCALL - * that enters idle (EXIT_REASON_HLT case). STI instruction enables - * interrupts only one instruction later. If there is a window between - * STI and the instruction that emulates the HALT state, there is a - * chance for interrupts to happen in this window, which can delay the - * HLT operation indefinitely. Since this is the not the desired - * result, conditionally call STI before TDCALL. - */ - testq $TDX_HCALL_ISSUE_STI, %rsi - jz .Lskip_sti - sti -.Lskip_sti: tdcall /*
arch/x86/coco/tdx/tdx.c +4 -19
@@ -169,7 +169,7 @@ static int ve_instr_len(struct ve_info * } } -static u64 __cpuidle __halt(const bool irq_disabled, const bool do_sti) +static u64 __cpuidle __halt(const bool irq_disabled) {struct tdx_hypercall_args args = { .r10 =TDX_HYPERCALL_STANDARD ,
@@ -189,20 +189,14 @@ static u64 __cpuidle __halt(const bool i* can keep the vCPU in virtual HLT, even if an IRQ is * pending, without hanging/breaking the guest. */ -return __tdx_hypercall (&args, do_sti ?TDX_HCALL_ISSUE_STI :0 ); +return __tdx_hypercall (&args,0 ); }static int handle_halt (struct ve_info *ve) { -/* -* Since non safe halt is mainly used in CPU offlining -* and the guest will always stay in the halt state, don't -* call the STI instruction (set do_sti as false). -*/ const bool irq_disabled =irqs_disabled (); -const bool do_sti = false; -if (__halt (irq_disabled, do_sti)) +if (__halt (irq_disabled))return -EIO ;return ve_instr_len (ve);
@@ -210,22 +204,13 @@ static int handle_halt(struct ve_info *v void __cpuidle tdx_safe_halt(void) { -/* -* For do_sti=true case, __tdx_hypercall() function enables -* interrupts using the STI instruction before the TDCALL. So -* set irq_disabled as false. -*/ const bool irq_disabled = false; -const bool do_sti = true;/* * Use WARN_ONCE() to report the failure. */ -if (__halt (irq_disabled, do_sti)) +if (__halt (irq_disabled))WARN_ONCE (1 ,"HLT instruction emulation failed\n" ); - -/* XXX I can't make sense of what @do_sti actually does */ -raw_local_irq_disable (); }static int read_msr (struct pt_regs *regs,struct ve_info *ve)
arch/x86/include/asm/shared/tdx.h +0 -1
@@ -8,7 +8,6 @@#define TDX_HYPERCALL_STANDARD 0#define TDX_HCALL_HAS_OUTPUT BIT(0) -#define TDX_HCALL_ISSUE_STI BIT(1)#define TDX_CPUID_LEAF_ID 0x21#define TDX_IDENT "IntelTDX "
[tip: sched/core] x86/tdx: Remove TDX_HCALL_ISSUE_STI
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e80a48bade619ec5a92230b3d4ae84bfc2746822 Gitweb: https://git.kernel.org/tip/e80a48bade619ec5a92230b3d4ae84bfc2746822 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:36 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
x86/tdx: Remove TDX_HCALL_ISSUE_STI
Now that arch_cpu_idle() is expected to return with IRQs disabled, avoid the useless STI/CLI dance.
Per the specs this is supposed to work, but nobody has yet relied up this behaviour so broken implementations are possible.
--- arch/x86/coco/tdx/tdcall.S | 13 ------------- arch/x86/coco/tdx/tdx.c | 23 ++++------------------- arch/x86/include/asm/shared/tdx.h | 1 - 3 files changed, 4 insertions(+), 33 deletions(-)
arch/x86/coco/tdx/tdcall.S +0 -13
@@ -139,19 +139,6 @@ SYM_FUNC_START(__tdx_hypercall) movl $TDVMCALL_EXPOSE_REGS_MASK, %ecx - /* - * For the idle loop STI needs to be called directly before the TDCALL - * that enters idle (EXIT_REASON_HLT case). STI instruction enables - * interrupts only one instruction later. If there is a window between - * STI and the instruction that emulates the HALT state, there is a - * chance for interrupts to happen in this window, which can delay the - * HLT operation indefinitely. Since this is the not the desired - * result, conditionally call STI before TDCALL. - */ - testq $TDX_HCALL_ISSUE_STI, %rsi - jz .Lskip_sti - sti -.Lskip_sti: tdcall /*
arch/x86/coco/tdx/tdx.c +4 -19
@@ -220,7 +220,7 @@ static int ve_instr_len(struct ve_info *ve) } } -static u64 __cpuidle __halt(const bool irq_disabled, const bool do_sti) +static u64 __cpuidle __halt(const bool irq_disabled) {struct tdx_hypercall_args args = { .r10 =TDX_HYPERCALL_STANDARD ,
@@ -240,20 +240,14 @@ static u64 __cpuidle __halt(const bool irq_disabled, const bool do_sti)* can keep the vCPU in virtual HLT, even if an IRQ is * pending, without hanging/breaking the guest. */ -return __tdx_hypercall (&args, do_sti ?TDX_HCALL_ISSUE_STI :0 ); +return __tdx_hypercall (&args,0 ); }static int handle_halt (struct ve_info *ve) { -/* -* Since non safe halt is mainly used in CPU offlining -* and the guest will always stay in the halt state, don't -* call the STI instruction (set do_sti as false). -*/ const bool irq_disabled =irqs_disabled (); -const bool do_sti = false; -if (__halt (irq_disabled, do_sti)) +if (__halt (irq_disabled))return -EIO ;return ve_instr_len (ve);
@@ -261,22 +255,13 @@ static int handle_halt(struct ve_info *ve) void __cpuidle tdx_safe_halt(void) { -/* -* For do_sti=true case, __tdx_hypercall() function enables -* interrupts using the STI instruction before the TDCALL. So -* set irq_disabled as false. -*/ const bool irq_disabled = false; -const bool do_sti = true;/* * Use WARN_ONCE() to report the failure. */ -if (__halt (irq_disabled, do_sti)) +if (__halt (irq_disabled))WARN_ONCE (1 ,"HLT instruction emulation failed\n" ); - -/* XXX I can't make sense of what @do_sti actually does */ -raw_local_irq_disable (); }static int read_msr (struct pt_regs *regs,struct ve_info *ve)
arch/x86/include/asm/shared/tdx.h +0 -1
@@ -8,7 +8,6 @@#define TDX_HYPERCALL_STANDARD 0#define TDX_HCALL_HAS_OUTPUT BIT(0) -#define TDX_HCALL_ISSUE_STI BIT(1)#define TDX_CPUID_LEAF_ID 0x21#define TDX_IDENT "IntelTDX "
Peter Zijlstra
maintainerPATCH v3 23/51 arm,smp: Remove trace_.*_rcuidle() usage
In reply to Peter ZijlstraNone of these functions should ever be ran with RCU disabled anymore.
Specifically, do_handle_IPI() is only called from handle_IPI() which explicitly does irq_enter()/irq_exit() which ensures RCU is watching.
The problem with smp_cross_call() was, per commit 7c64cc0531fa ("arm: Use _rcuidle for smp_cross_call() tracepoints"), that cpuidle_enter_state_coupled() already had RCU disabled, but that's long been fixed by commit 1098582a0f6c ("sched,idle,rcu: Push rcu_idle deeper into the idle path").
--- arch/arm/kernel/smp.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm/kernel/smp.c +3 -3
@@ -639,7 +639,7 @@ static void do_handle_IPI(int ipinr)unsigned int cpu =smp_processor_id ();if ((unsigned )ipinr <NR_IPI ) -trace_ipi_entry_rcuidle (ipi_types[ipinr]); +trace_ipi_entry (ipi_types[ipinr]);switch (ipinr) {case IPI_WAKEUP :
@@ -686,7 +686,7 @@ static void do_handle_IPI(int ipinr) }if ((unsigned )ipinr <NR_IPI ) -trace_ipi_exit_rcuidle (ipi_types[ipinr]); +trace_ipi_exit (ipi_types[ipinr]); }/* Legacy version, should go away once all irqchips have been converted */
@@ -709,7 +709,7 @@ static irqreturn_t ipi_handler(int irq,static void smp_cross_call (const struct cpumask *target,unsigned int ipinr) { -trace_ipi_raise_rcuidle (target, ipi_types[ipinr]); +trace_ipi_raise (target, ipi_types[ipinr]);__ipi_send_mask (ipi_desc[ipinr], target); }
[tip: sched/core] arm, smp: Remove trace_.*_rcuidle() usage
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 08a56e07cd3cb91dc700ba0a879d17e4879dd617 Gitweb: https://git.kernel.org/tip/08a56e07cd3cb91dc700ba0a879d17e4879dd617 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:37 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
arm, smp: Remove trace_.*_rcuidle() usage
None of these functions should ever be ran with RCU disabled anymore.
Specifically, do_handle_IPI() is only called from handle_IPI() which explicitly does irq_enter()/irq_exit() which ensures RCU is watching.
The problem with smp_cross_call() was, per commit description:
7c64cc0531fa ("arm: Use _rcuidle for smp_cross_call() tracepoints")
... that cpuidle_enter_state_coupled() already had RCU disabled, but that's long been fixed by commit:
1098582a0f6c ("sched,idle,rcu: Push rcu_idle deeper into the idle path")
--- arch/arm/kernel/smp.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm/kernel/smp.c +3 -3
@@ -638,7 +638,7 @@ static void do_handle_IPI(int ipinr)unsigned int cpu =smp_processor_id ();if ((unsigned )ipinr <NR_IPI ) -trace_ipi_entry_rcuidle (ipi_types[ipinr]); +trace_ipi_entry (ipi_types[ipinr]);switch (ipinr) {case IPI_WAKEUP :
@@ -685,7 +685,7 @@ static void do_handle_IPI(int ipinr) }if ((unsigned )ipinr <NR_IPI ) -trace_ipi_exit_rcuidle (ipi_types[ipinr]); +trace_ipi_exit (ipi_types[ipinr]); }/* Legacy version, should go away once all irqchips have been converted */
@@ -708,7 +708,7 @@ static irqreturn_t ipi_handler(int irq, void *data)static void smp_cross_call (const struct cpumask *target,unsigned int ipinr) { -trace_ipi_raise_rcuidle (target, ipi_types[ipinr]); +trace_ipi_raise (target, ipi_types[ipinr]);__ipi_send_mask (ipi_desc[ipinr], target); }
Peter Zijlstra
maintainerPATCH v3 24/51 arm64,smp: Remove trace_.*_rcuidle() usage
In reply to Peter ZijlstraEver since commit d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") this function is called in regular IRQ context.
--- arch/arm64/kernel/smp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/kernel/smp.c +2 -2
@@ -865,7 +865,7 @@ static void do_handle_IPI(int ipinr)unsigned int cpu =smp_processor_id ();if ((unsigned )ipinr <NR_IPI ) -trace_ipi_entry_rcuidle (ipi_types[ipinr]); +trace_ipi_entry (ipi_types[ipinr]);switch (ipinr) {case IPI_RESCHEDULE :
@@ -914,7 +914,7 @@ static void do_handle_IPI(int ipinr) }if ((unsigned )ipinr <NR_IPI ) -trace_ipi_exit_rcuidle (ipi_types[ipinr]); +trace_ipi_exit (ipi_types[ipinr]); }static irqreturn_t ipi_handler (int irq,void *data)
[tip: sched/core] arm64, smp: Remove trace_.*_rcuidle() usage
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4a3182e6d62c95f8572630c22aafee2306cfea58 Gitweb: https://git.kernel.org/tip/4a3182e6d62c95f8572630c22aafee2306cfea58 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:38 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:15 +01:00
arm64, smp: Remove trace_.*_rcuidle() usage
Ever since commit d3afc7f12987 ("arm64: Allow IPIs to be handled as normal interrupts") this function is called in regular IRQ context.
--- arch/arm64/kernel/smp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/arm64/kernel/smp.c +2 -2
@@ -865,7 +865,7 @@ static void do_handle_IPI(int ipinr)unsigned int cpu =smp_processor_id ();if ((unsigned )ipinr <NR_IPI ) -trace_ipi_entry_rcuidle (ipi_types[ipinr]); +trace_ipi_entry (ipi_types[ipinr]);switch (ipinr) {case IPI_RESCHEDULE :
@@ -914,7 +914,7 @@ static void do_handle_IPI(int ipinr) }if ((unsigned )ipinr <NR_IPI ) -trace_ipi_exit_rcuidle (ipi_types[ipinr]); +trace_ipi_exit (ipi_types[ipinr]); }static irqreturn_t ipi_handler (int irq,void *data)
Peter Zijlstra
maintainerPATCH v3 25/51 printk: Remove trace_.*_rcuidle() usage
In reply to Peter ZijlstraThe problem, per commit fc98c3c8c9dc ("printk: use rcuidle console tracepoint"), was printk usage from the cpuidle path where RCU was already disabled.
Per the patches earlier in this series, this is no longer the case.
--- kernel/printk/printk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/printk/printk.c +1 -1
@@ -2238,7 +2238,7 @@ static u16 printk_sprint(char *text, u16 } } -trace_console_rcuidle (text, text_len); +trace_console (text, text_len);return text_len; }
[tip: sched/core] printk: Remove trace_.*_rcuidle() usage
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 880970b56b732b158385991b232592144256f330 Gitweb: https://git.kernel.org/tip/880970b56b732b158385991b232592144256f330 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:39 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
printk: Remove trace_.*_rcuidle() usage
The problem, per commit fc98c3c8c9dc ("printk: use rcuidle console tracepoint"), was printk usage from the cpuidle path where RCU was already disabled.
Per the patches earlier in this series, this is no longer the case.
--- kernel/printk/printk.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/printk/printk.c +1 -1
@@ -2194,7 +2194,7 @@ static u16 printk_sprint(char *text, u16 size, int facility, } } -trace_console_rcuidle (text, text_len); +trace_console (text, text_len);return text_len; }
Peter Zijlstra
maintainerPATCH v3 26/51 time/tick-broadcast: Remove RCU_NONIDLE usage
In reply to Peter ZijlstraNo callers left that have already disabled RCU.
--- kernel/time/tick-broadcast-hrtimer.c | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-)
kernel/time/tick-broadcast-hrtimer.c +12 -17
@@ -56,25 +56,20 @@ static int bc_set_next(ktime_t expires,* hrtimer callback function is currently running, then * hrtimer_start() cannot move it and the timer stays on the CPU on * which it is assigned at the moment. +*/ +hrtimer_start (&bctimer, expires,HRTIMER_MODE_ABS_PINNED_HARD ); +/* +* The core tick broadcast mode expects bc->bound_on to be set +* correctly to prevent a CPU which has the broadcast hrtimer +* armed from going deep idle. * -* As this can be called from idle code, the hrtimer_start() -* invocation has to be wrapped with RCU_NONIDLE() as -* hrtimer_start() can call into tracing. +* As tick_broadcast_lock is held, nothing can change the cpu +* base which was just established in hrtimer_start() above. So +* the below access is safe even without holding the hrtimer +* base lock. */ -RCU_NONIDLE ( { -hrtimer_start (&bctimer, expires,HRTIMER_MODE_ABS_PINNED_HARD ); -/* -* The core tick broadcast mode expects bc->bound_on to be set -* correctly to prevent a CPU which has the broadcast hrtimer -* armed from going deep idle. -* -* As tick_broadcast_lock is held, nothing can change the cpu -* base which was just established in hrtimer_start() above. So -* the below access is safe even without holding the hrtimer -* base lock. -*/ - bc->bound_on = bctimer.base->cpu_base->cpu; - } ); + bc->bound_on = bctimer.base->cpu_base->cpu; +return 0 ; }
[tip: sched/core] time/tick-broadcast: Remove RCU_NONIDLE() usage
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e3ee5e66f78e9950b9ada276700abd95e9b144d7 Gitweb: https://git.kernel.org/tip/e3ee5e66f78e9950b9ada276700abd95e9b144d7 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:40 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
time/tick-broadcast: Remove RCU_NONIDLE() usage
No callers left that have already disabled RCU.
--- kernel/time/tick-broadcast-hrtimer.c | 29 +++++++++++---------------- 1 file changed, 12 insertions(+), 17 deletions(-)
kernel/time/tick-broadcast-hrtimer.c +12 -17
@@ -56,25 +56,20 @@ static int bc_set_next(ktime_t expires, struct clock_event_device *bc)* hrtimer callback function is currently running, then * hrtimer_start() cannot move it and the timer stays on the CPU on * which it is assigned at the moment. +*/ +hrtimer_start (&bctimer, expires,HRTIMER_MODE_ABS_PINNED_HARD ); +/* +* The core tick broadcast mode expects bc->bound_on to be set +* correctly to prevent a CPU which has the broadcast hrtimer +* armed from going deep idle. * -* As this can be called from idle code, the hrtimer_start() -* invocation has to be wrapped with RCU_NONIDLE() as -* hrtimer_start() can call into tracing. +* As tick_broadcast_lock is held, nothing can change the cpu +* base which was just established in hrtimer_start() above. So +* the below access is safe even without holding the hrtimer +* base lock. */ -RCU_NONIDLE ( { -hrtimer_start (&bctimer, expires,HRTIMER_MODE_ABS_PINNED_HARD ); -/* -* The core tick broadcast mode expects bc->bound_on to be set -* correctly to prevent a CPU which has the broadcast hrtimer -* armed from going deep idle. -* -* As tick_broadcast_lock is held, nothing can change the cpu -* base which was just established in hrtimer_start() above. So -* the below access is safe even without holding the hrtimer -* base lock. -*/ - bc->bound_on = bctimer.base->cpu_base->cpu; - } ); + bc->bound_on = bctimer.base->cpu_base->cpu; +return 0 ; }
Peter Zijlstra
maintainerPATCH v3 27/51 cpuidle,sched: Remove annotations from TIF_{POLLING_NRFLAG,NEED_RESCHED}
In reply to Peter Zijlstravmlinux.o: warning: objtool: mwait_idle+0x5: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0xc5: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0xbc: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xea: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xb4: call to current_set_polling_and_test() leaves .noinstr.text section
vmlinux.o: warning: objtool: intel_idle+0xa6: call to current_clr_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xbf: call to current_clr_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xa1: call to current_clr_polling() leaves .noinstr.text section
vmlinux.o: warning: objtool: mwait_idle+0xe: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0xc5: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0xbc: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xea: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xb4: call to __current_set_polling() leaves .noinstr.text section
vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0x73: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0x91: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0x78: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_safe_halt+0xf: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section
--- include/linux/sched/idle.h | 40 ++++++++++++++++++++++++++++++---------- include/linux/thread_info.h | 18 +++++++++++++++++- 2 files changed, 47 insertions(+), 11 deletions(-)
include/linux/sched/idle.h +30 -10
@@ -23,12 +23,37 @@ static inline void wake_up_if_idle(int c*/ #ifdef TIF_POLLING_NRFLAG -static inline void __current_set_polling (void ) +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_ATOMIC_H + +static __always_inline void __current_set_polling(void) { -set_thread_flag (TIF_POLLING_NRFLAG ); +arch_set_bit (TIF_POLLING_NRFLAG , + (unsigned long *)(¤t_thread_info ()->flags)); } -static inline bool __must_check current_set_polling_and_test (void ) +static __always_inline void __current_clr_polling(void) +{ +arch_clear_bit (TIF_POLLING_NRFLAG , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +#else + +static __always_inline void __current_set_polling(void) +{ +set_bit (TIF_POLLING_NRFLAG , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +static __always_inline void __current_clr_polling(void) +{ +clear_bit (TIF_POLLING_NRFLAG , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_ATOMIC_H */ + +static __always_inline bool __must_check current_set_polling_and_test(void) {__current_set_polling ();
@@ -41,12 +66,7 @@ static inline bool __must_check current_return unlikely (tif_need_resched ()); } -static inline void __current_clr_polling (void ) -{ -clear_thread_flag (TIF_POLLING_NRFLAG ); -} - -static inline bool __must_check current_clr_polling_and_test (void ) +static __always_inline bool __must_check current_clr_polling_and_test(void) {__current_clr_polling ();
@@ -73,7 +93,7 @@ static inline bool __must_check current_ }#endif -static inline void current_clr_polling (void ) +static __always_inline void current_clr_polling(void) {__current_clr_polling ();
include/linux/thread_info.h +17 -1
@@ -177,7 +177,23 @@ static __always_inline unsigned long rea clear_ti_thread_flag(task_thread_info(t), TIF_##fl)#endif /* !CONFIG_GENERIC_ENTRY */ -#define tif_need_resched () test_thread_flag(TIF_NEED_RESCHED) +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H + +static __always_inline bool tif_need_resched(void) +{ +return arch_test_bit (TIF_NEED_RESCHED , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +#else + +static __always_inline bool tif_need_resched(void) +{ +return test_bit (TIF_NEED_RESCHED , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H */ #ifndef CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES static inline int arch_within_stack_frames(const void * const stack,
[tip: sched/core] cpuidle, sched: Remove instrumentation from TIF_{POLLING_NRFLAG,NEED_RESCHED}
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: e4df1511e1f4bcaa0d590aa7bbffe8bbbd6dfb49 Gitweb: https://git.kernel.org/tip/e4df1511e1f4bcaa0d590aa7bbffe8bbbd6dfb49 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:41 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
cpuidle, sched: Remove instrumentation from TIF_{POLLING_NRFLAG,NEED_RESCHED}
objtool pointed out that various idle-TIF management methods have instrumentation:
vmlinux.o: warning: objtool: mwait_idle+0x5: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0xc5: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0xbc: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xea: call to current_set_polling_and_test() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xb4: call to current_set_polling_and_test() leaves .noinstr.text section
vmlinux.o: warning: objtool: intel_idle+0xa6: call to current_clr_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xbf: call to current_clr_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xa1: call to current_clr_polling() leaves .noinstr.text section
vmlinux.o: warning: objtool: mwait_idle+0xe: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0xc5: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0xbc: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xea: call to __current_set_polling() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0xb4: call to __current_set_polling() leaves .noinstr.text section
vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x73: call to test_ti_thread_flag() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0x73: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0x91: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0x78: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_safe_halt+0xf: call to test_ti_thread_flag.constprop.0() leaves .noinstr.text section
Remove the instrumentation, because these methods are used in low-level cpuidle code moving between states, that should not be instrumented.
--- include/linux/sched/idle.h | 40 +++++++++++++++++++++++++++--------- include/linux/thread_info.h | 18 +++++++++++++++- 2 files changed, 47 insertions(+), 11 deletions(-)
include/linux/sched/idle.h +30 -10
@@ -23,12 +23,37 @@ static inline void wake_up_if_idle(int cpu) { }*/ #ifdef TIF_POLLING_NRFLAG -static inline void __current_set_polling (void ) +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_ATOMIC_H + +static __always_inline void __current_set_polling(void) { -set_thread_flag (TIF_POLLING_NRFLAG ); +arch_set_bit (TIF_POLLING_NRFLAG , + (unsigned long *)(¤t_thread_info ()->flags)); } -static inline bool __must_check current_set_polling_and_test (void ) +static __always_inline void __current_clr_polling(void) +{ +arch_clear_bit (TIF_POLLING_NRFLAG , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +#else + +static __always_inline void __current_set_polling(void) +{ +set_bit (TIF_POLLING_NRFLAG , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +static __always_inline void __current_clr_polling(void) +{ +clear_bit (TIF_POLLING_NRFLAG , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_ATOMIC_H */ + +static __always_inline bool __must_check current_set_polling_and_test(void) {__current_set_polling ();
@@ -41,12 +66,7 @@ static inline bool __must_check current_set_polling_and_test(void)return unlikely (tif_need_resched ()); } -static inline void __current_clr_polling (void ) -{ -clear_thread_flag (TIF_POLLING_NRFLAG ); -} - -static inline bool __must_check current_clr_polling_and_test (void ) +static __always_inline bool __must_check current_clr_polling_and_test(void) {__current_clr_polling ();
@@ -73,7 +93,7 @@ static inline bool __must_check current_clr_polling_and_test(void) }#endif -static inline void current_clr_polling (void ) +static __always_inline void current_clr_polling(void) {__current_clr_polling ();
include/linux/thread_info.h +17 -1
@@ -177,7 +177,23 @@ static __always_inline unsigned long read_ti_thread_flags(struct thread_info *ti clear_ti_thread_flag(task_thread_info(t), TIF_##fl)#endif /* !CONFIG_GENERIC_ENTRY */ -#define tif_need_resched () test_thread_flag(TIF_NEED_RESCHED) +#ifdef _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H + +static __always_inline bool tif_need_resched(void) +{ +return arch_test_bit (TIF_NEED_RESCHED , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +#else + +static __always_inline bool tif_need_resched(void) +{ +return test_bit (TIF_NEED_RESCHED , + (unsigned long *)(¤t_thread_info ()->flags)); +} + +#endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_NON_ATOMIC_H */ #ifndef CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES static inline int arch_within_stack_frames(const void * const stack,
Peter Zijlstra
maintainerPATCH v3 28/51 cpuidle,mwait: Make noinstr clean
In reply to Peter Zijlstravmlinux.o: warning: objtool: intel_idle_s2idle+0x6e: call to __monitor.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0x8c: call to __monitor.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0x73: call to __monitor.constprop.0() leaves .noinstr.text section
vmlinux.o: warning: objtool: mwait_idle+0x88: call to clflush() leaves .noinstr.text section
--- arch/x86/include/asm/mwait.h | 12 ++++++------ arch/x86/include/asm/special_insns.h | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-)
arch/x86/include/asm/mwait.h +6 -6
@@ -25,7 +25,7 @@#define TPAUSE_C01_STATE 1#define TPAUSE_C02_STATE 0 -static inline void __monitor (const void *eax,unsigned long ecx, +static __always_inline void __monitor(const void *eax, unsigned long ecx,unsigned long edx) {/* "monitor %eax, %ecx, %edx;" */
@@ -33,7 +33,7 @@ static inline void __monitor(const void :: "a" (eax), "c" (ecx), "d"(edx)); } -static inline void __monitorx (const void *eax,unsigned long ecx, +static __always_inline void __monitorx(const void *eax, unsigned long ecx,unsigned long edx) {/* "monitorx %eax, %ecx, %edx;" */
@@ -41,7 +41,7 @@ static inline void __monitorx(const void :: "a" (eax), "c" (ecx), "d"(edx)); } -static inline void __mwait (unsigned long eax,unsigned long ecx) +static __always_inline void __mwait(unsigned long eax, unsigned long ecx) {mds_idle_clear_cpu_buffers ();
@@ -76,8 +76,8 @@ static inline void __mwait(unsigned long* EAX (logical) address to monitor * ECX #GP if not zero */ -static inline void __mwaitx (unsigned long eax,unsigned long ebx, -unsigned long ecx) +static __always_inline void __mwaitx(unsigned long eax, unsigned long ebx, +unsigned long ecx) {/* No MDS buffer clear as this is AMD/HYGON only */
@@ -86,7 +86,7 @@ static inline void __mwaitx(unsigned lon :: "a" (eax), "b" (ebx), "c" (ecx)); } -static inline void __sti_mwait (unsigned long eax,unsigned long ecx) +static __always_inline void __sti_mwait(unsigned long eax, unsigned long ecx) {mds_idle_clear_cpu_buffers ();/* "mwait %eax, %ecx;" */
arch/x86/include/asm/special_insns.h +1 -1
@@ -196,7 +196,7 @@ static inline void load_gs_index(unsigne#endif /* CONFIG_PARAVIRT_XXL */ -static inline void clflush (volatile void *__p) +static __always_inline void clflush(volatile void *__p) { asmvolatile ("clflush %0" :"+m" (*(volatile char __force *)__p)); }
[tip: sched/core] cpuidle, mwait: Make the mwait code noinstr clean
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 2ec8efe64e6dfcaa50906ae4b0b5a0a7e1a54d50 Gitweb: https://git.kernel.org/tip/2ec8efe64e6dfcaa50906ae4b0b5a0a7e1a54d50 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:42 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
cpuidle, mwait: Make the mwait code noinstr clean
objtool found a few cases where this code called out into instrumented code:
vmlinux.o: warning: objtool: intel_idle_s2idle+0x6e: call to __monitor.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0x8c: call to __monitor.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0x73: call to __monitor.constprop.0() leaves .noinstr.text section
vmlinux.o: warning: objtool: mwait_idle+0x88: call to clflush() leaves .noinstr.text section
Fix it by marking the affected methods as __always_inline.
--- arch/x86/include/asm/mwait.h | 12 ++++++------ arch/x86/include/asm/special_insns.h | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-)
arch/x86/include/asm/mwait.h +6 -6
@@ -26,7 +26,7 @@#define TPAUSE_C01_STATE 1#define TPAUSE_C02_STATE 0 -static inline void __monitor (const void *eax,unsigned long ecx, +static __always_inline void __monitor(const void *eax, unsigned long ecx,unsigned long edx) {/* "monitor %eax, %ecx, %edx;" */
@@ -34,7 +34,7 @@ static inline void __monitor(const void *eax, unsigned long ecx, :: "a" (eax), "c" (ecx), "d"(edx)); } -static inline void __monitorx (const void *eax,unsigned long ecx, +static __always_inline void __monitorx(const void *eax, unsigned long ecx,unsigned long edx) {/* "monitorx %eax, %ecx, %edx;" */
@@ -42,7 +42,7 @@ static inline void __monitorx(const void *eax, unsigned long ecx, :: "a" (eax), "c" (ecx), "d"(edx)); } -static inline void __mwait (unsigned long eax,unsigned long ecx) +static __always_inline void __mwait(unsigned long eax, unsigned long ecx) {mds_idle_clear_cpu_buffers ();
@@ -77,8 +77,8 @@ static inline void __mwait(unsigned long eax, unsigned long ecx)* EAX (logical) address to monitor * ECX #GP if not zero */ -static inline void __mwaitx (unsigned long eax,unsigned long ebx, -unsigned long ecx) +static __always_inline void __mwaitx(unsigned long eax, unsigned long ebx, +unsigned long ecx) {/* No MDS buffer clear as this is AMD/HYGON only */
@@ -87,7 +87,7 @@ static inline void __mwaitx(unsigned long eax, unsigned long ebx, :: "a" (eax), "b" (ebx), "c" (ecx)); } -static inline void __sti_mwait (unsigned long eax,unsigned long ecx) +static __always_inline void __sti_mwait(unsigned long eax, unsigned long ecx) {mds_idle_clear_cpu_buffers ();/* "mwait %eax, %ecx;" */
arch/x86/include/asm/special_insns.h +1 -1
@@ -196,7 +196,7 @@ static inline void load_gs_index(unsigned int selector)#endif /* CONFIG_PARAVIRT_XXL */ -static inline void clflush (volatile void *__p) +static __always_inline void clflush(volatile void *__p) { asmvolatile ("clflush %0" :"+m" (*(volatile char __force *)__p)); }
Peter Zijlstra
maintainerPATCH v3 29/51 cpuidle,tdx: Make tdx noinstr clean
In reply to Peter Zijlstravmlinux.o: warning: objtool: __halt+0x2c: call to hcall_func.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: __halt+0x3f: call to __tdx_hypercall() leaves .noinstr.text section vmlinux.o: warning: objtool: __tdx_hypercall+0x66: call to __tdx_hypercall_failed() leaves .noinstr.text section
--- arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/coco/tdx/tdcall.S | 2 ++ arch/x86/coco/tdx/tdx.c | 5 +++-- 3 files changed, 6 insertions(+), 2 deletions(-)
arch/x86/boot/compressed/vmlinux.lds.S +1 -0
@@ -34,6 +34,7 @@ SECTIONS _text = .; /* Text */ *(.text) *(.text.*) + *(.noinstr.text) _etext = . ; } .rodata : {
arch/x86/coco/tdx/tdcall.S +2 -0
@@ -31,6 +31,8 @@ TDX_R12 | TDX_R13 | \ TDX_R14 | TDX_R15 ) +.section .noinstr.text, "ax" + /* * __tdx_module_call() - Used by TDX guests to request services from * the TDX module (does not include VMM services) using TDCALL instruction.
arch/x86/coco/tdx/tdx.c +3 -2
@@ -53,8 +53,9 @@ static inline u64 _tdx_hypercall(u64 fn, }/* Called from __tdx_hypercall() for unrecoverable failure */ -void __tdx_hypercall_failed (void ) +noinstr void __tdx_hypercall_failed(void) { +instrumentation_begin ();panic ("TDVMCALL failed. TDX module bug?" ); }
@@ -64,7 +65,7 @@ void __tdx_hypercall_failed(void)* Reusing the KVM EXIT_REASON macros makes it easier to connect the host and * guest sides of these calls. */ -static u64 hcall_func (u64 exit_reason) +static __always_inline u64 hcall_func(u64 exit_reason) {return exit_reason; }
[tip: sched/core] cpuidle, tdx: Make TDX code noinstr clean
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: c3982c1a36f70bd4521a9852246ea50105912084 Gitweb: https://git.kernel.org/tip/c3982c1a36f70bd4521a9852246ea50105912084 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:43 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
cpuidle, tdx: Make TDX code noinstr clean
objtool found a few cases where this code called out into instrumented code:
vmlinux.o: warning: objtool: __halt+0x2c: call to hcall_func.constprop.0() leaves .noinstr.text section vmlinux.o: warning: objtool: __halt+0x3f: call to __tdx_hypercall() leaves .noinstr.text section vmlinux.o: warning: objtool: __tdx_hypercall+0x66: call to __tdx_hypercall_failed() leaves .noinstr.text section
Fix it by:
- moving TDX tdcall assembly methods into .noinstr.text (they are already noistr-clean)
- marking __tdx_hypercall_failed() as 'noinstr'
- annotating hcall_func() as __always_inline
--- arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/coco/tdx/tdcall.S | 2 ++ arch/x86/coco/tdx/tdx.c | 5 +++-- 3 files changed, 6 insertions(+), 2 deletions(-)
arch/x86/boot/compressed/vmlinux.lds.S +1 -0
@@ -34,6 +34,7 @@ SECTIONS _text = .; /* Text */ *(.text) *(.text.*) + *(.noinstr.text) _etext = . ; } .rodata : {
arch/x86/coco/tdx/tdcall.S +2 -0
@@ -31,6 +31,8 @@ TDX_R12 | TDX_R13 | \ TDX_R14 | TDX_R15 ) +.section .noinstr.text, "ax" + /* * __tdx_module_call() - Used by TDX guests to request services from * the TDX module (does not include VMM services) using TDCALL instruction.
arch/x86/coco/tdx/tdx.c +3 -2
@@ -64,8 +64,9 @@ static inline u64 _tdx_hypercall(u64 fn, u64 r12, u64 r13, u64 r14, u64 r15) }/* Called from __tdx_hypercall() for unrecoverable failure */ -void __tdx_hypercall_failed (void ) +noinstr void __tdx_hypercall_failed(void) { +instrumentation_begin ();panic ("TDVMCALL failed. TDX module bug?" ); }
@@ -75,7 +76,7 @@ void __tdx_hypercall_failed(void)* Reusing the KVM EXIT_REASON macros makes it easier to connect the host and * guest sides of these calls. */ -static u64 hcall_func (u64 exit_reason) +static __always_inline u64 hcall_func(u64 exit_reason) {return exit_reason; }
Peter Zijlstra
maintainerPATCH v3 30/51 cpuidle,xenpv: Make more PARAVIRT_XXL noinstr clean
In reply to Peter Zijlstravmlinux.o: warning: objtool: acpi_idle_enter_s2idle+0xde: call to wbinvd() leaves .noinstr.text section vmlinux.o: warning: objtool: default_idle+0x4: call to arch_safe_halt() leaves .noinstr.text section vmlinux.o: warning: objtool: xen_safe_halt+0xa: call to HYPERVISOR_sched_op.constprop.0() leaves .noinstr.text section
--- arch/x86/include/asm/paravirt.h | 6 ++++-- arch/x86/include/asm/special_insns.h | 4 ++-- arch/x86/include/asm/xen/hypercall.h | 2 +- arch/x86/kernel/paravirt.c | 14 ++++++++++++-- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/irq.c | 2 +- 6 files changed, 21 insertions(+), 9 deletions(-)
arch/x86/include/asm/paravirt.h +4 -2
@@ -168,7 +168,7 @@ static inline void __write_cr4(unsignedPVOP_VCALL1 (cpu.write_cr4, x); } -static inline void arch_safe_halt (void ) +static __always_inline void arch_safe_halt(void) {PVOP_VCALL0 (irq.safe_halt); }
@@ -178,7 +178,9 @@ static inline void halt(void)PVOP_VCALL0 (irq.halt); } -static inline void wbinvd (void ) +extern noinstr void pv_native_wbinvd(void); + +static __always_inline void wbinvd(void) {PVOP_ALT_VCALL0 (cpu.wbinvd,"wbinvd" ,ALT_NOT (X86_FEATURE_XENPV )); }
arch/x86/include/asm/special_insns.h +2 -2
@@ -115,7 +115,7 @@ static inline void wrpkru(u32 pkru) }#endif -static inline void native_wbinvd (void ) +static __always_inline void native_wbinvd(void) { asmvolatile ("wbinvd" : : :"memory" ); }
@@ -179,7 +179,7 @@ static inline void __write_cr4(unsignednative_write_cr4 (x); } -static inline void wbinvd (void ) +static __always_inline void wbinvd(void) {native_wbinvd (); }
arch/x86/include/asm/xen/hypercall.h +1 -1
@@ -382,7 +382,7 @@ MULTI_stack_switch(struct multicall_entr }#endif -static inline int +static __always_inline intHYPERVISOR_sched_op (int cmd,void *arg) {return _hypercall2 (int, sched_op, cmd, arg);
arch/x86/kernel/paravirt.c +12 -2
@@ -233,6 +233,11 @@ static noinstr void pv_native_set_debugrnative_set_debugreg (regno, val); } +noinstr void pv_native_wbinvd(void) +{ +native_wbinvd (); +} + static noinstr void pv_native_irq_enable(void) {native_irq_enable ();
@@ -242,6 +247,11 @@ static noinstr void pv_native_irq_disabl {native_irq_disable (); } + +static noinstr void pv_native_safe_halt(void) +{ +native_safe_halt (); +}#endif enum paravirt_lazy_mode paravirt_get_lazy_mode (void )
@@ -273,7 +283,7 @@ struct paravirt_patch_template pv_ops = .cpu.read_cr0 = native_read_cr0, .cpu.write_cr0 = native_write_cr0, .cpu.write_cr4 = native_write_cr4, - .cpu.wbinvd = native_wbinvd, + .cpu.wbinvd = pv_native_wbinvd, .cpu.read_msr = native_read_msr, .cpu.write_msr = native_write_msr, .cpu.read_msr_safe = native_read_msr_safe,
@@ -307,7 +317,7 @@ struct paravirt_patch_template pv_ops = .irq.save_fl = __PV_IS_CALLEE_SAVE(native_save_fl), .irq.irq_disable = __PV_IS_CALLEE_SAVE(pv_native_irq_disable), .irq.irq_enable = __PV_IS_CALLEE_SAVE(pv_native_irq_enable), - .irq.safe_halt = native_safe_halt, + .irq.safe_halt = pv_native_safe_halt, .irq.halt = native_halt,#endif /* CONFIG_PARAVIRT_XXL */
arch/x86/xen/enlighten_pv.c +1 -1
@@ -1019,7 +1019,7 @@ static const typeof(pv_ops) xen_cpu_ops .write_cr4 = xen_write_cr4, - .wbinvd = native_wbinvd, + .wbinvd = pv_native_wbinvd, .read_msr = xen_read_msr, .write_msr = xen_write_msr,
arch/x86/xen/irq.c +1 -1
@@ -24,7 +24,7 @@ noinstr void xen_force_evtchn_callback(v (void )HYPERVISOR_xen_version (0 ,NULL ); } -static void xen_safe_halt (void ) +static noinstr void xen_safe_halt(void) {/* Blocking includes an implicit local_irq_enable(). */ if (HYPERVISOR_sched_op (SCHEDOP_block,NULL ) !=0 )
[tip: sched/core] cpuidle, xenpv: Make more PARAVIRT_XXL noinstr clean
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 10a099405fdf96c3205ae775abf19fcd301b31b4 Gitweb: https://git.kernel.org/tip/10a099405fdf96c3205ae775abf19fcd301b31b4 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:44 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
cpuidle, xenpv: Make more PARAVIRT_XXL noinstr clean
objtool found a few cases where this code called out into instrumented code:
vmlinux.o: warning: objtool: acpi_idle_enter_s2idle+0xde: call to wbinvd() leaves .noinstr.text section vmlinux.o: warning: objtool: default_idle+0x4: call to arch_safe_halt() leaves .noinstr.text section vmlinux.o: warning: objtool: xen_safe_halt+0xa: call to HYPERVISOR_sched_op.constprop.0() leaves .noinstr.text section
Solve this by:
- marking arch_safe_halt(), wbinvd(), native_wbinvd() and HYPERVISOR_sched_op() as __always_inline().
- Explicitly uninlining xen_safe_halt() and pv_native_wbinvd() [they were already uninlined by the compiler on use as function pointers] and annotating them as 'noinstr'.
- Annotating pv_native_safe_halt() as 'noinstr'.
--- arch/x86/include/asm/paravirt.h | 6 ++++-- arch/x86/include/asm/special_insns.h | 4 ++-- arch/x86/include/asm/xen/hypercall.h | 2 +- arch/x86/kernel/paravirt.c | 14 ++++++++++++-- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/irq.c | 2 +- 6 files changed, 21 insertions(+), 9 deletions(-)
arch/x86/include/asm/paravirt.h +4 -2
@@ -168,7 +168,7 @@ static inline void __write_cr4(unsigned long x)PVOP_VCALL1 (cpu.write_cr4, x); } -static inline void arch_safe_halt (void ) +static __always_inline void arch_safe_halt(void) {PVOP_VCALL0 (irq.safe_halt); }
@@ -178,7 +178,9 @@ static inline void halt(void)PVOP_VCALL0 (irq.halt); } -static inline void wbinvd (void ) +extern noinstr void pv_native_wbinvd(void); + +static __always_inline void wbinvd(void) {PVOP_ALT_VCALL0 (cpu.wbinvd,"wbinvd" ,ALT_NOT (X86_FEATURE_XENPV )); }
arch/x86/include/asm/special_insns.h +2 -2
@@ -115,7 +115,7 @@ static inline void wrpkru(u32 pkru) }#endif -static inline void native_wbinvd (void ) +static __always_inline void native_wbinvd(void) { asmvolatile ("wbinvd" : : :"memory" ); }
@@ -179,7 +179,7 @@ static inline void __write_cr4(unsigned long x)native_write_cr4 (x); } -static inline void wbinvd (void ) +static __always_inline void wbinvd(void) {native_wbinvd (); }
arch/x86/include/asm/xen/hypercall.h +1 -1
@@ -382,7 +382,7 @@ MULTI_stack_switch(struct multicall_entry *mcl, }#endif -static inline int +static __always_inline intHYPERVISOR_sched_op (int cmd,void *arg) {return _hypercall2 (int, sched_op, cmd, arg);
arch/x86/kernel/paravirt.c +12 -2
@@ -216,6 +216,11 @@ static noinstr void pv_native_set_debugreg(int regno, unsigned long val)native_set_debugreg (regno, val); } +noinstr void pv_native_wbinvd(void) +{ +native_wbinvd (); +} + static noinstr void pv_native_irq_enable(void) {native_irq_enable ();
@@ -225,6 +230,11 @@ static noinstr void pv_native_irq_disable(void) {native_irq_disable (); } + +static noinstr void pv_native_safe_halt(void) +{ +native_safe_halt (); +}#endif enum paravirt_lazy_mode paravirt_get_lazy_mode (void )
@@ -256,7 +266,7 @@ struct paravirt_patch_template pv_ops = { .cpu.read_cr0 = native_read_cr0, .cpu.write_cr0 = native_write_cr0, .cpu.write_cr4 = native_write_cr4, - .cpu.wbinvd = native_wbinvd, + .cpu.wbinvd = pv_native_wbinvd, .cpu.read_msr = native_read_msr, .cpu.write_msr = native_write_msr, .cpu.read_msr_safe = native_read_msr_safe,
@@ -290,7 +300,7 @@ struct paravirt_patch_template pv_ops = { .irq.save_fl = __PV_IS_CALLEE_SAVE(native_save_fl), .irq.irq_disable = __PV_IS_CALLEE_SAVE(pv_native_irq_disable), .irq.irq_enable = __PV_IS_CALLEE_SAVE(pv_native_irq_enable), - .irq.safe_halt = native_safe_halt, + .irq.safe_halt = pv_native_safe_halt, .irq.halt = native_halt,#endif /* CONFIG_PARAVIRT_XXL */
arch/x86/xen/enlighten_pv.c +1 -1
@@ -1068,7 +1068,7 @@ static const typeof(pv_ops) xen_cpu_ops __initconst = { .write_cr4 = xen_write_cr4, - .wbinvd = native_wbinvd, + .wbinvd = pv_native_wbinvd, .read_msr = xen_read_msr, .write_msr = xen_write_msr,
arch/x86/xen/irq.c +1 -1
@@ -24,7 +24,7 @@ noinstr void xen_force_evtchn_callback(void) (void )HYPERVISOR_xen_version (0 ,NULL ); } -static void xen_safe_halt (void ) +static noinstr void xen_safe_halt(void) {/* Blocking includes an implicit local_irq_enable(). */ if (HYPERVISOR_sched_op (SCHEDOP_block,NULL ) !=0 )
Peter Zijlstra
maintainerPATCH v3 31/51 cpuidle,nospec: Make noinstr clean
In reply to Peter Zijlstravmlinux.o: warning: objtool: mwait_idle+0x47: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0xa2: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0x91: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0x8c: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xaa: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section
--- arch/x86/include/asm/nospec-branch.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/nospec-branch.h +1 -1
@@ -310,7 +310,7 @@ static __always_inline void mds_user_cle* * Clear CPU buffers if the corresponding static key is enabled */ -static inline void mds_idle_clear_cpu_buffers (void ) +static __always_inline void mds_idle_clear_cpu_buffers(void) {if (static_branch_likely (&mds_idle_clear))mds_clear_cpu_buffers ();
[tip: sched/core] cpuidle, nospec: Make mds_idle_clear_cpu_buffers() noinstr clean
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 10fdb38cee36b8dc4775e6fe4ad85e78bafdbd65 Gitweb: https://git.kernel.org/tip/10fdb38cee36b8dc4775e6fe4ad85e78bafdbd65 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:45 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
cpuidle, nospec: Make mds_idle_clear_cpu_buffers() noinstr clean
objtool found that the mds_idle_clear_cpu_buffers() method got uninlined by the compiler where it called out into instrumentation:
vmlinux.o: warning: objtool: mwait_idle+0x47: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0xa2: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle+0x91: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_s2idle+0x8c: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0xaa: call to mds_idle_clear_cpu_buffers() leaves .noinstr.text section
Solve this by marking mds_idle_clear_cpu_buffers() as __always_inline.
--- arch/x86/include/asm/nospec-branch.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/nospec-branch.h +1 -1
@@ -564,7 +564,7 @@ static __always_inline void mds_user_clear_cpu_buffers(void)* * Clear CPU buffers if the corresponding static key is enabled */ -static inline void mds_idle_clear_cpu_buffers (void ) +static __always_inline void mds_idle_clear_cpu_buffers(void) {if (static_branch_likely (&mds_idle_clear))mds_clear_cpu_buffers ();
Peter Zijlstra
maintainerPATCH v3 32/51 cpuidle,acpi: Make noinstr clean
In reply to Peter Zijlstravmlinux.o: warning: objtool: io_idle+0xc: call to __inb.isra.0() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_idle_enter+0xfe: call to num_online_cpus() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_idle_enter+0x115: call to acpi_idle_fallback_to_c1.isra.0() leaves .noinstr.text section
--- arch/x86/include/asm/shared/io.h | 4 ++-- drivers/acpi/processor_idle.c | 2 +- include/linux/cpumask.h | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/shared/io.h +2 -2
@@ -5,13 +5,13 @@#include <linux/types.h> #define BUILDIO (bwl, bw, type) \ -static inline void __out##bwl(type value, u16 port) \ +static __always_inline void __out##bwl(type value, u16 port) \ { \ asm volatile("out" #bwl " %" #bw "0, %w1" \ : : "a"(value), "Nd"(port)); \ } \ \ -static inline type __in##bwl(u16 port) \ +static __always_inline type __in##bwl(u16 port) \ { \ type value; \ asm volatile("in" #bwl " %w1, %" #bw "0" \
drivers/acpi/processor_idle.c +1 -1
@@ -593,7 +593,7 @@ static int acpi_idle_play_dead(struct cpreturn 0 ; } -static bool acpi_idle_fallback_to_c1 (struct acpi_processor *pr) +static __always_inline bool acpi_idle_fallback_to_c1(struct acpi_processor *pr) {return IS_ENABLED (CONFIG_HOTPLUG_CPU ) && !pr->flags.has_cst && !(acpi_gbl_FADT.flags &ACPI_FADT_C2_MP_SUPPORTED );
include/linux/cpumask.h +2 -2
@@ -908,9 +908,9 @@ static inline const struct cpumask *get_* concurrent CPU hotplug operations unless invoked from a cpuhp_lock held * region. */ -static inline unsigned int num_online_cpus (void ) +static __always_inline unsigned int num_online_cpus(void) { -return atomic_read (&__num_online_cpus); +return arch_atomic_read (&__num_online_cpus); }#define num_possible_cpus () cpumask_weight(cpu_possible_mask)#define num_present_cpus () cpumask_weight(cpu_present_mask)
[tip: sched/core] cpuidle, ACPI: Make noinstr clean
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 6a123d6ae6ea930b9bb3c595ceac2b2f93039f67 Gitweb: https://git.kernel.org/tip/6a123d6ae6ea930b9bb3c595ceac2b2f93039f67 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:46 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
cpuidle, ACPI: Make noinstr clean
objtool found cases where ACPI methods called out into instrumentation code:
vmlinux.o: warning: objtool: io_idle+0xc: call to __inb.isra.0() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_idle_enter+0xfe: call to num_online_cpus() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_idle_enter+0x115: call to acpi_idle_fallback_to_c1.isra.0() leaves .noinstr.text section
Fix this by: marking the IO in/out, acpi_idle_fallback_to_c1() and num_online_cpus() methods as __always_inline.
--- arch/x86/include/asm/shared/io.h | 4 ++-- drivers/acpi/processor_idle.c | 2 +- include/linux/cpumask.h | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/include/asm/shared/io.h +2 -2
@@ -5,13 +5,13 @@#include <linux/types.h> #define BUILDIO (bwl, bw, type) \ -static inline void __out##bwl(type value, u16 port) \ +static __always_inline void __out##bwl(type value, u16 port) \ { \ asm volatile("out" #bwl " %" #bw "0, %w1" \ : : "a"(value), "Nd"(port)); \ } \ \ -static inline type __in##bwl(u16 port) \ +static __always_inline type __in##bwl(u16 port) \ { \ type value; \ asm volatile("in" #bwl " %w1, %" #bw "0" \
drivers/acpi/processor_idle.c +1 -1
@@ -607,7 +607,7 @@ static int acpi_idle_play_dead(struct cpuidle_device *dev, int index)return 0 ; } -static bool acpi_idle_fallback_to_c1 (struct acpi_processor *pr) +static __always_inline bool acpi_idle_fallback_to_c1(struct acpi_processor *pr) {return IS_ENABLED (CONFIG_HOTPLUG_CPU ) && !pr->flags.has_cst && !(acpi_gbl_FADT.flags &ACPI_FADT_C2_MP_SUPPORTED );
include/linux/cpumask.h +2 -2
@@ -1017,9 +1017,9 @@ static inline const struct cpumask *get_cpu_mask(unsigned int cpu)* concurrent CPU hotplug operations unless invoked from a cpuhp_lock held * region. */ -static inline unsigned int num_online_cpus (void ) +static __always_inline unsigned int num_online_cpus(void) { -return atomic_read (&__num_online_cpus); +return arch_atomic_read (&__num_online_cpus); }#define num_possible_cpus () cpumask_weight(cpu_possible_mask)#define num_present_cpus () cpumask_weight(cpu_present_mask)
Peter Zijlstra
maintainerPATCH v3 33/51 trace: Remove trace_hardirqs_{on,off}_caller()
In reply to Peter ZijlstraPer commit 56e62a737028 ("s390: convert to generic entry") the last and only callers of trace_hardirqs_{on,off}_caller() went away, clean up.
--- kernel/trace/trace_preemptirq.c | 29 ----------------------------- 1 file changed, 29 deletions(-)
kernel/trace/trace_preemptirq.c +0 -29
@@ -84,35 +84,6 @@ void trace_hardirqs_off(void) }EXPORT_SYMBOL (trace_hardirqs_off);NOKPROBE_SYMBOL (trace_hardirqs_off); - -__visible void trace_hardirqs_on_caller(unsigned long caller_addr) -{ -if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 , caller_addr); -tracer_hardirqs_on (CALLER_ADDR0 , caller_addr); -this_cpu_write (tracing_irq_cpu,0 ); - } - -lockdep_hardirqs_on_prepare (); -lockdep_hardirqs_on (caller_addr); -} -EXPORT_SYMBOL (trace_hardirqs_on_caller); -NOKPROBE_SYMBOL (trace_hardirqs_on_caller); - -__visible void trace_hardirqs_off_caller(unsigned long caller_addr) -{ -lockdep_hardirqs_off (caller_addr); - -if (!this_cpu_read (tracing_irq_cpu)) { -this_cpu_write (tracing_irq_cpu,1 ); -tracer_hardirqs_off (CALLER_ADDR0 , caller_addr); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 , caller_addr); - } -} -EXPORT_SYMBOL (trace_hardirqs_off_caller); -NOKPROBE_SYMBOL (trace_hardirqs_off_caller);#endif /* CONFIG_TRACE_IRQFLAGS */ #ifdef CONFIG_TRACE_PREEMPT_TOGGLE
[tip: sched/core] tracing: Remove trace_hardirqs_{on,off}_caller()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: dc7305606d480f34d9ed960fcaceac81d0f38436 Gitweb: https://git.kernel.org/tip/dc7305606d480f34d9ed960fcaceac81d0f38436 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:47 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
tracing: Remove trace_hardirqs_{on,off}_caller()
Per commit 56e62a737028 ("s390: convert to generic entry") the last and only callers of trace_hardirqs_{on,off}_caller() went away, clean up.
--- kernel/trace/trace_preemptirq.c | 29 ----------------------------- 1 file changed, 29 deletions(-)
kernel/trace/trace_preemptirq.c +0 -29
@@ -84,35 +84,6 @@ void trace_hardirqs_off(void) }EXPORT_SYMBOL (trace_hardirqs_off);NOKPROBE_SYMBOL (trace_hardirqs_off); - -__visible void trace_hardirqs_on_caller(unsigned long caller_addr) -{ -if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 , caller_addr); -tracer_hardirqs_on (CALLER_ADDR0 , caller_addr); -this_cpu_write (tracing_irq_cpu,0 ); - } - -lockdep_hardirqs_on_prepare (); -lockdep_hardirqs_on (caller_addr); -} -EXPORT_SYMBOL (trace_hardirqs_on_caller); -NOKPROBE_SYMBOL (trace_hardirqs_on_caller); - -__visible void trace_hardirqs_off_caller(unsigned long caller_addr) -{ -lockdep_hardirqs_off (caller_addr); - -if (!this_cpu_read (tracing_irq_cpu)) { -this_cpu_write (tracing_irq_cpu,1 ); -tracer_hardirqs_off (CALLER_ADDR0 , caller_addr); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 , caller_addr); - } -} -EXPORT_SYMBOL (trace_hardirqs_off_caller); -NOKPROBE_SYMBOL (trace_hardirqs_off_caller);#endif /* CONFIG_TRACE_IRQFLAGS */ #ifdef CONFIG_TRACE_PREEMPT_TOGGLE
Peter Zijlstra
maintainerPATCH v3 34/51 trace: WARN on rcuidle
In reply to Peter ZijlstraARCH_WANTS_NO_INSTR (a superset of CONFIG_GENERIC_ENTRY) disallows any and all tracing when RCU isn't enabled.
--- include/linux/tracepoint.h | 15 +++++++++++++-- kernel/trace/trace.c | 3 +++ 2 files changed, 16 insertions(+), 2 deletions(-)
include/linux/tracepoint.h +13 -2
@@ -178,6 +178,17 @@ static inline struct tracepoint *tracepo#endif /* CONFIG_HAVE_STATIC_CALL */ /* +* ARCH_WANTS_NO_INSTR archs are expected to have sanitized entry and idle +* code that disallow any/all tracing/instrumentation when RCU isn't watching. +*/ +#ifdef CONFIG_ARCH_WANTS_NO_INSTR +#define RCUIDLE_COND (rcuidle) (rcuidle) +#else +/* srcu can't be used from NMI */ +#define RCUIDLE_COND (rcuidle) (rcuidle && in_nmi()) +#endif + +/* * it_func[0] is never NULL because there is at least one element in the array * when the array itself is non NULL. */
@@ -188,8 +199,8 @@ static inline struct tracepoint *tracepoif (!(cond)) \return ; \ \ -/* srcu can't be used from NMI */ \ -WARN_ON_ONCE (rcuidle &&in_nmi ()); \ +if (WARN_ON_ONCE (RCUIDLE_COND (rcuidle))) \ +return ; \ \/* keep srcu and sched-rcu usage consistent */ \preempt_disable_notrace (); \
kernel/trace/trace.c +3 -0
@@ -3119,6 +3119,9 @@ void __trace_stack(struct trace_array *treturn ; } +if (WARN_ON_ONCE (IS_ENABLED (CONFIG_GENERIC_ENTRY ))) +return ; +/* * When an NMI triggers, RCU is enabled via ct_nmi_enter(), * but if the above rcu_is_watching() failed, then the NMI
[tip: sched/core] tracing: WARN on rcuidle
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 408b961146be4c1a776ce285c3c289afab15298a Gitweb: https://git.kernel.org/tip/408b961146be4c1a776ce285c3c289afab15298a Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:48 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
tracing: WARN on rcuidle
ARCH_WANTS_NO_INSTR (a superset of CONFIG_GENERIC_ENTRY) disallows any and all tracing when RCU isn't enabled.
--- include/linux/tracepoint.h | 15 +++++++++++++-- kernel/trace/trace.c | 3 +++ 2 files changed, 16 insertions(+), 2 deletions(-)
include/linux/tracepoint.h +13 -2
@@ -178,6 +178,17 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)#endif /* CONFIG_HAVE_STATIC_CALL */ /* +* ARCH_WANTS_NO_INSTR archs are expected to have sanitized entry and idle +* code that disallow any/all tracing/instrumentation when RCU isn't watching. +*/ +#ifdef CONFIG_ARCH_WANTS_NO_INSTR +#define RCUIDLE_COND (rcuidle) (rcuidle) +#else +/* srcu can't be used from NMI */ +#define RCUIDLE_COND (rcuidle) (rcuidle && in_nmi()) +#endif + +/* * it_func[0] is never NULL because there is at least one element in the array * when the array itself is non NULL. */
@@ -188,8 +199,8 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)if (!(cond)) \return ; \ \ -/* srcu can't be used from NMI */ \ -WARN_ON_ONCE (rcuidle &&in_nmi ()); \ +if (WARN_ON_ONCE (RCUIDLE_COND (rcuidle))) \ +return ; \ \/* keep srcu and sched-rcu usage consistent */ \preempt_disable_notrace (); \
kernel/trace/trace.c +3 -0
@@ -3128,6 +3128,9 @@ void __trace_stack(struct trace_array *tr, unsigned int trace_ctx,return ; } +if (WARN_ON_ONCE (IS_ENABLED (CONFIG_GENERIC_ENTRY ))) +return ; +/* * When an NMI triggers, RCU is enabled via ct_nmi_enter(), * but if the above rcu_is_watching() failed, then the NMI
Peter Zijlstra
maintainerPATCH v3 35/51 trace,hardirq: No moar _rcuidle() tracing
In reply to Peter ZijlstraRobot reported that trace_hardirqs_{on,off}() tickle the forbidden _rcuidle() tracepoint through local_irq_{en,dis}able().
For 'sane' configs, these calls will only happen with RCU enabled and as such can use the regular tracepoint. This also means it's possible to trace them from NMI context again.
--- kernel/trace/trace_preemptirq.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
kernel/trace/trace_preemptirq.c +13 -8
@@ -20,6 +20,15 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);/* +* ... +*/ +#ifdef CONFIG_ARCH_WANTS_NO_INSTR +#define trace (point) trace_##point +#else +#define trace (point) if (!in_nmi()) trace_##point##_rcuidle +#endif + +/* * Like trace_hardirqs_on() but without the lockdep invocation. This is * used in the low level entry code where the ordering vs. RCU is important * and lockdep uses a staged approach which splits the lockdep hardirq
@@ -28,8 +37,7 @@ static DEFINE_PER_CPU(int, tracing_irq_cvoid trace_hardirqs_on_prepare (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }
@@ -40,8 +48,7 @@ NOKPROBE_SYMBOL(trace_hardirqs_on_preparvoid trace_hardirqs_on (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }
@@ -63,8 +70,7 @@ void trace_hardirqs_off_finish(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }
@@ -78,8 +84,7 @@ void trace_hardirqs_off(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }EXPORT_SYMBOL (trace_hardirqs_off);
[tip: sched/core] tracing, hardirq: No moar _rcuidle() tracing
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 9aedeaed6fc6fe8452b9b8225e95cc2b8631ff91 Gitweb: https://git.kernel.org/tip/9aedeaed6fc6fe8452b9b8225e95cc2b8631ff91 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:49 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
tracing, hardirq: No moar _rcuidle() tracing
Robot reported that trace_hardirqs_{on,off}() tickle the forbidden _rcuidle() tracepoint through local_irq_{en,dis}able().
For 'sane' configs, these calls will only happen with RCU enabled and as such can use the regular tracepoint. This also means it's possible to trace them from NMI context again.
--- kernel/trace/trace_preemptirq.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)
kernel/trace/trace_preemptirq.c +18 -8
@@ -20,6 +20,20 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);/* +* Use regular trace points on architectures that implement noinstr +* tooling: these calls will only happen with RCU enabled, which can +* use a regular tracepoint. +* +* On older architectures, use the rcuidle tracing methods (which +* aren't NMI-safe - so exclude NMI contexts): +*/ +#ifdef CONFIG_ARCH_WANTS_NO_INSTR +#define trace (point) trace_##point +#else +#define trace (point) if (!in_nmi()) trace_##point##_rcuidle +#endif + +/* * Like trace_hardirqs_on() but without the lockdep invocation. This is * used in the low level entry code where the ordering vs. RCU is important * and lockdep uses a staged approach which splits the lockdep hardirq
@@ -28,8 +42,7 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);void trace_hardirqs_on_prepare (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }
@@ -40,8 +53,7 @@ NOKPROBE_SYMBOL(trace_hardirqs_on_prepare);void trace_hardirqs_on (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }
@@ -63,8 +75,7 @@ void trace_hardirqs_off_finish(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }
@@ -78,8 +89,7 @@ void trace_hardirqs_off(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }EXPORT_SYMBOL (trace_hardirqs_off);
Masami Hiramatsu (Google)
maintainerRe: [PATCH v3 35/51] trace,hardirq: No moar _rcuidle() tracing
In reply to Peter ZijlstraHi Peter,
Quoted 8 lines · On Thu, 12 Jan 2023 20:43:49 +0100 Peter Zijlstra <[email protected]> wrote:
Robot reported that trace_hardirqs_{on,off}() tickle the forbidden
_rcuidle()tracepoint through local_irq_{en,dis}able().For 'sane' configs, these calls will only happen with RCU enabled and as such can use the regular tracepoint. This also means it's possible to trace them from NMI context again.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
The code looks good to me. I just have a question about comment.
kernel/trace/trace_preemptirq.c line 23
Quoted 11 lines
--- kernel/trace/trace_preemptirq.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)kernel/trace/trace_preemptirq.c +1
-0@@ -20,6 +20,15 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);/* +* ...
Is this intended? Wouldn't you leave any comment here?
Thank you,
Quoted 53 lines
kernel/trace/trace_preemptirq.c +12
-8+*/ +#ifdef CONFIG_ARCH_WANTS_NO_INSTR +#define trace (point) trace_##point +#else +#define trace (point) if (!in_nmi()) trace_##point##_rcuidle +#endif + +/* * Like trace_hardirqs_on() but without the lockdep invocation. This is * used in the low level entry code where the ordering vs. RCU is important * and lockdep uses a staged approach which splits the lockdep hardirq @@ -28,8 +37,7 @@ static DEFINE_PER_CPU(int, tracing_irq_cvoid trace_hardirqs_on_prepare (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }@@ -40,8 +48,7 @@ NOKPROBE_SYMBOL(trace_hardirqs_on_preparvoid trace_hardirqs_on (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }@@ -63,8 +70,7 @@ void trace_hardirqs_off_finish(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }@@ -78,8 +84,7 @@ void trace_hardirqs_off(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }EXPORT_SYMBOL (trace_hardirqs_off);
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Peter Zijlstra
maintainerRe: [PATCH v3 35/51] trace,hardirq: No moar _rcuidle() tracing
Quoted 25 lines · Masami Hiramatsu
Hi Peter,
Quoted 8 lines · On Thu, 12 Jan 2023 20:43:49 +0100 Peter Zijlstra <[email protected]> wrote:
Robot reported that trace_hardirqs_{on,off}() tickle the forbidden
_rcuidle()tracepoint through local_irq_{en,dis}able().For 'sane' configs, these calls will only happen with RCU enabled and as such can use the regular tracepoint. This also means it's possible to trace them from NMI context again.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>The code looks good to me. I just have a question about comment.
Quoted 9 lines
--- kernel/trace/trace_preemptirq.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)kernel/trace/trace_preemptirq.c +0
-0@@ -20,6 +20,15 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);
Quoted 4 lines
kernel/trace/trace_preemptirq.c line 23
Quoted 2 lines
kernel/trace/trace_preemptirq.c +1
-0/* +* ... Is this intended? Wouldn't you leave any comment here?
I indeed forgot to write the comment before posting, my bad :/ Ingo fixed it up when he applied.
Ingo Molnar
maintainerPATCH v4 1/1 trace,hardirq: No moar _rcuidle() tracing
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Masami Hiramatsu
Hi Peter,
Quoted 8 lines · On Thu, 12 Jan 2023 20:43:49 +0100 Peter Zijlstra <[email protected]> wrote:
Robot reported that trace_hardirqs_{on,off}() tickle the forbidden
_rcuidle()tracepoint through local_irq_{en,dis}able().For 'sane' configs, these calls will only happen with RCU enabled and as such can use the regular tracepoint. This also means it's possible to trace them from NMI context again.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>The code looks good to me. I just have a question about comment.
Quoted 9 lines
--- kernel/trace/trace_preemptirq.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)kernel/trace/trace_preemptirq.c +0
-0@@ -20,6 +20,15 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);
Quoted 7 lines
Quoted 4 lines
kernel/trace/trace_preemptirq.c line 23
Quoted 2 lines
kernel/trace/trace_preemptirq.c +1
-0/* +* ... Is this intended? Wouldn't you leave any comment here?
I indeed forgot to write the comment before posting, my bad :/ Ingo fixed it up when he applied.
For completeness, I've attached the final commit, which has this comment included:
+/*
+ * Use regular trace points on architectures that implement noinstr
+ * tooling: these calls will only happen with RCU enabled, which can
+ * use a regular tracepoint.
+ *
+ * On older architectures, use the rcuidle tracing methods (which
+ * aren't NMI-safe - so exclude NMI contexts):
+ */
Thanks,
Ingo
================>
From: Peter Zijlstra <[email protected]>
Date: Thu, 12 Jan 2023 20:43:49 +0100
Subject: [PATCH] tracing, hardirq: No moar _rcuidle() tracing
Robot reported that trace_hardirqs_{on,off}() tickle the forbidden _rcuidle() tracepoint through local_irq_{en,dis}able().
For 'sane' configs, these calls will only happen with RCU enabled and as such can use the regular tracepoint. This also means it's possible to trace them from NMI context again.
--- kernel/trace/trace_preemptirq.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)
kernel/trace/trace_preemptirq.c +18 -8
@@ -19,6 +19,20 @@/* Per-cpu variable to prevent redundant calls when IRQs already off */ static DEFINE_PER_CPU(int, tracing_irq_cpu); +/* +* Use regular trace points on architectures that implement noinstr +* tooling: these calls will only happen with RCU enabled, which can +* use a regular tracepoint. +* +* On older architectures, use the rcuidle tracing methods (which +* aren't NMI-safe - so exclude NMI contexts): +*/ +#ifdef CONFIG_ARCH_WANTS_NO_INSTR +#define trace (point) trace_##point +#else +#define trace (point) if (!in_nmi()) trace_##point##_rcuidle +#endif +/* * Like trace_hardirqs_on() but without the lockdep invocation. This is * used in the low level entry code where the ordering vs. RCU is important
@@ -28,8 +42,7 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);void trace_hardirqs_on_prepare (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }
@@ -40,8 +53,7 @@ NOKPROBE_SYMBOL(trace_hardirqs_on_prepare);void trace_hardirqs_on (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }
@@ -63,8 +75,7 @@ void trace_hardirqs_off_finish(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }
@@ -78,8 +89,7 @@ void trace_hardirqs_off(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }EXPORT_SYMBOL (trace_hardirqs_off);
Re: [PATCH v4] trace,hardirq: No moar _rcuidle() tracing
Quoted 40 lines · On Tue, 17 Jan 2023 10:17:16 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Masami Hiramatsu
Hi Peter,
Quoted 8 lines · On Thu, 12 Jan 2023 20:43:49 +0100 Peter Zijlstra <[email protected]> wrote:
Robot reported that trace_hardirqs_{on,off}() tickle the forbidden
_rcuidle()tracepoint through local_irq_{en,dis}able().For 'sane' configs, these calls will only happen with RCU enabled and as such can use the regular tracepoint. This also means it's possible to trace them from NMI context again.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>The code looks good to me. I just have a question about comment.
Quoted 9 lines
--- kernel/trace/trace_preemptirq.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)kernel/trace/trace_preemptirq.c +0
-0@@ -20,6 +20,15 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);Quoted 7 lines
Quoted 4 lines
kernel/trace/trace_preemptirq.c line 23
Quoted 2 lines
kernel/trace/trace_preemptirq.c +1
-0/* +* ... Is this intended? Wouldn't you leave any comment here?
I indeed forgot to write the comment before posting, my bad :/ Ingo fixed it up when he applied.
For completeness, I've attached the final commit, which has this comment included:
Quoted 8 lines
+/*
+ * Use regular trace points on architectures that implement noinstr
+ * tooling: these calls will only happen with RCU enabled, which can
+ * use a regular tracepoint.
+ *
+ * On older architectures, use the rcuidle tracing methods (which
+ * aren't NMI-safe - so exclude NMI contexts):
+ */
Thanks! This looks good to me.
Quoted 90 lines
Thanks,
Ingo
================>
From: Peter Zijlstra <[email protected]>
Date: Thu, 12 Jan 2023 20:43:49 +0100
Subject: [PATCH] tracing, hardirq: No moar_rcuidle()tracingRobot reported that trace_hardirqs_{on,off}() tickle the forbidden
_rcuidle()tracepoint through local_irq_{en,dis}able().For 'sane' configs, these calls will only happen with RCU enabled and as such can use the regular tracepoint. This also means it's possible to trace them from NMI context again.
Reported-by kernel test robot <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Ingo Molnar <[email protected]>--- kernel/trace/trace_preemptirq.c | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-)kernel/trace/trace_preemptirq.c +18
-8@@ -19,6 +19,20 @@/* Per-cpu variable to prevent redundant calls when IRQs already off */ static DEFINE_PER_CPU(int, tracing_irq_cpu); +/* +* Use regular trace points on architectures that implement noinstr +* tooling: these calls will only happen with RCU enabled, which can +* use a regular tracepoint. +* +* On older architectures, use the rcuidle tracing methods (which +* aren't NMI-safe - so exclude NMI contexts): +*/ +#ifdef CONFIG_ARCH_WANTS_NO_INSTR +#define trace (point) trace_##point +#else +#define trace (point) if (!in_nmi()) trace_##point##_rcuidle +#endif +/* * Like trace_hardirqs_on() but without the lockdep invocation. This is * used in the low level entry code where the ordering vs. RCU is important @@ -28,8 +42,7 @@ static DEFINE_PER_CPU(int, tracing_irq_cpu);void trace_hardirqs_on_prepare (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }@@ -40,8 +53,7 @@ NOKPROBE_SYMBOL(trace_hardirqs_on_prepare);void trace_hardirqs_on (void ) {if (this_cpu_read (tracing_irq_cpu)) { -if (!in_nmi ()) -trace_irq_enable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_enable)(CALLER_ADDR0 ,CALLER_ADDR1 );tracer_hardirqs_on (CALLER_ADDR0 ,CALLER_ADDR1 );this_cpu_write (tracing_irq_cpu,0 ); }@@ -63,8 +75,7 @@ void trace_hardirqs_off_finish(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }@@ -78,8 +89,7 @@ void trace_hardirqs_off(void)if (!this_cpu_read (tracing_irq_cpu)) {this_cpu_write (tracing_irq_cpu,1 );tracer_hardirqs_off (CALLER_ADDR0 ,CALLER_ADDR1 ); -if (!in_nmi ()) -trace_irq_disable_rcuidle (CALLER_ADDR0 ,CALLER_ADDR1 ); +trace (irq_disable)(CALLER_ADDR0 ,CALLER_ADDR1 ); } }EXPORT_SYMBOL (trace_hardirqs_off);
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Peter Zijlstra
maintainerPATCH v3 36/51 cpuidle,omap3: Use WFI for omap3_pm_idle()
In reply to Peter Zijlstraarch_cpu_idle() is a very simple idle interface and exposes only a single idle state and is expected to not require RCU and not do any tracing/instrumentation.
As such, omap_sram_idle() is not a valid implementation. Replace it with the simple (shallow) omap3_do_wfi() call. Leaving the more complicated idle states for the cpuidle driver.
--- arch/arm/mach-omap2/pm34xx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/mach-omap2/pm34xx.c +1 -1
@@ -294,7 +294,7 @@ static void omap3_pm_idle(void)if (omap_irq_pending ())return ; -omap_sram_idle (); +omap3_do_wfi (); }#ifdef CONFIG_SUSPEND
[tip: sched/core] cpuidle, OMAP3: Use WFI for omap3_pm_idle()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 40dbea961a3d92a7dec92a72c18ee03d33ccc52a Gitweb: https://git.kernel.org/tip/40dbea961a3d92a7dec92a72c18ee03d33ccc52a Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:50 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
cpuidle, OMAP3: Use WFI for omap3_pm_idle()
arch_cpu_idle() is a very simple idle interface and exposes only a single idle state and is expected to not require RCU and not do any tracing/instrumentation.
As such, omap_sram_idle() is not a valid implementation. Replace it with the simple (shallow) omap3_do_wfi() call. Leaving the more complicated idle states for the cpuidle driver.
--- arch/arm/mach-omap2/pm34xx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/mach-omap2/pm34xx.c +1 -1
@@ -294,7 +294,7 @@ static void omap3_pm_idle(void)if (omap_irq_pending ())return ; -omap_sram_idle (); +omap3_do_wfi (); }#ifdef CONFIG_SUSPEND
Peter Zijlstra
maintainerPATCH v3 37/51 cpuidle,omap3: Push RCU-idle into omap_sram_idle()
In reply to Peter ZijlstraOMAP3 uses full SoC suspend modes as idle states, as such it needs the whole power-domain and clock-domain code from the idle path.
All that code is not suitable to run with RCU disabled, as such push RCU-idle deeper still.
--- arch/arm/mach-omap2/cpuidle34xx.c | 4 +--- arch/arm/mach-omap2/pm.h | 2 +- arch/arm/mach-omap2/pm34xx.c | 12 ++++++++++-- 3 files changed, 12 insertions(+), 6 deletions(-)
arch/arm/mach-omap2/cpuidle34xx.c +1 -3
@@ -133,9 +133,7 @@ static int omap3_enter_idle(struct cpuid }/* Execute ARM wfi */ -ct_cpuidle_enter (); -omap_sram_idle (); -ct_cpuidle_exit (); +omap_sram_idle (true);/* * Call idle CPU PM enter notifier chain to restore
arch/arm/mach-omap2/pm.h +1 -1
@@ -29,7 +29,7 @@ static inline int omap4_idle_init(void)extern void *omap3_secure_ram_storage;extern void omap3_pm_off_mode_enable (int ); -extern void omap_sram_idle (void ); +extern void omap_sram_idle (bool rcuidle);extern int omap_pm_clkdms_setup (struct clockdomain *clkdm,void *unused);#if defined(CONFIG_PM_OPP )
arch/arm/mach-omap2/pm34xx.c +10 -2
@@ -26,6 +26,7 @@#include <linux/delay.h> #include <linux/slab.h> #include <linux/of.h> +#include <linux/cpuidle.h> #include <trace/events/power.h>
@@ -174,7 +175,7 @@ static int omap34xx_do_sram_idle(unsignereturn 0 ; } -void omap_sram_idle (void ) +void omap_sram_idle (bool rcuidle) {/* Variable to tell what needs to be saved and restored * in omap_sram_idle*/
@@ -254,11 +255,18 @@ void omap_sram_idle(void)*/ if (save_state)omap34xx_save_context (omap3_arm_context); + +if (rcuidle) +ct_cpuidle_enter (); +if (save_state ==1 || save_state ==3 )cpu_suspend (save_state, omap34xx_do_sram_idle);else omap34xx_do_sram_idle (save_state); +if (rcuidle) +ct_cpuidle_exit (); +/* Restore normal SDRC POWER settings */ if (cpu_is_omap3430() && omap_rev() >= OMAP3430_REV_ES3_0 && (omap_type() == OMAP2_DEVICE_TYPE_EMU ||
@@ -316,7 +324,7 @@ static int omap3_pm_suspend(void)omap3_intc_suspend (); -omap_sram_idle (); +omap_sram_idle (false); restore:/* Restore next_pwrsts */
[tip: sched/core] cpuidle, OMAP3: Push RCU-idle into omap_sram_idle()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 8c0956aa76f97003e2ac96e86b00a2a4e82abf8e Gitweb: https://git.kernel.org/tip/8c0956aa76f97003e2ac96e86b00a2a4e82abf8e Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:51 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:16 +01:00
cpuidle, OMAP3: Push RCU-idle into omap_sram_idle()
OMAP3 uses full SoC suspend modes as idle states, as such it needs the whole power-domain and clock-domain code from the idle path.
All that code is not suitable to run with RCU disabled, as such push RCU-idle deeper still.
--- arch/arm/mach-omap2/cpuidle34xx.c | 4 +--- arch/arm/mach-omap2/pm.h | 2 +- arch/arm/mach-omap2/pm34xx.c | 12 ++++++++++-- 3 files changed, 12 insertions(+), 6 deletions(-)
arch/arm/mach-omap2/cpuidle34xx.c +1 -3
@@ -133,9 +133,7 @@ static int omap3_enter_idle(struct cpuidle_device *dev, }/* Execute ARM wfi */ -ct_cpuidle_enter (); -omap_sram_idle (); -ct_cpuidle_exit (); +omap_sram_idle (true);/* * Call idle CPU PM enter notifier chain to restore
arch/arm/mach-omap2/pm.h +1 -1
@@ -29,7 +29,7 @@ static inline int omap4_idle_init(void)extern void *omap3_secure_ram_storage;extern void omap3_pm_off_mode_enable (int ); -extern void omap_sram_idle (void ); +extern void omap_sram_idle (bool rcuidle);extern int omap_pm_clkdms_setup (struct clockdomain *clkdm,void *unused);#if defined(CONFIG_PM_OPP )
arch/arm/mach-omap2/pm34xx.c +10 -2
@@ -26,6 +26,7 @@#include <linux/delay.h> #include <linux/slab.h> #include <linux/of.h> +#include <linux/cpuidle.h> #include <trace/events/power.h>
@@ -174,7 +175,7 @@ static int omap34xx_do_sram_idle(unsigned long save_state)return 0 ; } -void omap_sram_idle (void ) +void omap_sram_idle (bool rcuidle) {/* Variable to tell what needs to be saved and restored * in omap_sram_idle*/
@@ -254,11 +255,18 @@ void omap_sram_idle(void)*/ if (save_state)omap34xx_save_context (omap3_arm_context); + +if (rcuidle) +ct_cpuidle_enter (); +if (save_state ==1 || save_state ==3 )cpu_suspend (save_state, omap34xx_do_sram_idle);else omap34xx_do_sram_idle (save_state); +if (rcuidle) +ct_cpuidle_exit (); +/* Restore normal SDRC POWER settings */ if (cpu_is_omap3430() && omap_rev() >= OMAP3430_REV_ES3_0 && (omap_type() == OMAP2_DEVICE_TYPE_EMU ||
@@ -316,7 +324,7 @@ static int omap3_pm_suspend(void)omap3_intc_suspend (); -omap_sram_idle (); +omap_sram_idle (false); restore:/* Restore next_pwrsts */
Peter Zijlstra
maintainerPATCH v3 38/51 cpuidle,omap4: Push RCU-idle into omap4_enter_lowpower()
In reply to Peter ZijlstraFrom: Tony Lindgren <[email protected]>
OMAP4 uses full SoC suspend modes as idle states, as such it needs the whole power-domain and clock-domain code from the idle path.
All that code is not suitable to run with RCU disabled, as such push RCU-idle deeper still.
--- arch/arm/mach-omap2/common.h | 6 ++++-- arch/arm/mach-omap2/cpuidle44xx.c | 8 ++------ arch/arm/mach-omap2/omap-mpuss-lowpower.c | 12 +++++++++++- arch/arm/mach-omap2/pm44xx.c | 2 +- 4 files changed, 18 insertions(+), 10 deletions(-)
arch/arm/mach-omap2/common.h +4 -2
@@ -284,11 +284,13 @@ extern u32 omap4_get_cpu1_ns_pa_addr(voi#if defined(CONFIG_SMP ) && defined(CONFIG_PM )extern int omap4_mpuss_init (void ); -extern int omap4_enter_lowpower (unsigned int cpu,unsigned int power_state); +extern int omap4_enter_lowpower (unsigned int cpu,unsigned int power_state, +bool rcuidle);extern int omap4_hotplug_cpu (unsigned int cpu,unsigned int power_state);#else static inline int omap4_enter_lowpower (unsigned int cpu, -unsigned int power_state) +unsigned int power_state, +bool rcuidle) {cpu_do_idle ();return 0 ;
arch/arm/mach-omap2/cpuidle44xx.c +2 -6
@@ -105,9 +105,7 @@ static int omap_enter_idle_smp(struct cp }raw_spin_unlock_irqrestore (&mpu_lock, flag); -ct_cpuidle_enter (); -omap4_enter_lowpower (dev->cpu, cx->cpu_state); -ct_cpuidle_exit (); +omap4_enter_lowpower (dev->cpu, cx->cpu_state, true);raw_spin_lock_irqsave (&mpu_lock, flag);if (cx->mpu_state_vote ==num_online_cpus ())
@@ -186,10 +184,8 @@ static int omap_enter_idle_coupled(struc } } -ct_cpuidle_enter (); -omap4_enter_lowpower (dev->cpu, cx->cpu_state); +omap4_enter_lowpower (dev->cpu, cx->cpu_state, true); cpu_done[dev->cpu] = true; -ct_cpuidle_exit ();/* Wakeup CPU1 only if it is not offlined */ if (dev->cpu ==0 &&cpumask_test_cpu (1 , cpu_online_mask)) {
arch/arm/mach-omap2/omap-mpuss-lowpower.c +11 -1
@@ -33,6 +33,7 @@* and first to wake-up when MPUSS low power states are excercised */ +#include <linux/cpuidle.h> #include <linux/kernel.h> #include <linux/io.h> #include <linux/errno.h>
@@ -214,6 +215,7 @@ static void __init save_l2x0_context(voi* of OMAP4 MPUSS subsystem * @cpu : CPU ID * @power_state: Low power state. +* @rcuidle: RCU needs to be idled * * MPUSS states for the context save: * save_state =
@@ -222,7 +224,8 @@ static void __init save_l2x0_context(voi* 2 - CPUx L1 and logic lost + GIC lost: MPUSS OSWR * 3 - CPUx L1 and logic lost + GIC + L2 lost: DEVICE OFF */ -int omap4_enter_lowpower (unsigned int cpu,unsigned int power_state) +int omap4_enter_lowpower (unsigned int cpu,unsigned int power_state, +bool rcuidle) {struct omap4_cpu_pm_info *pm_info = &per_cpu (omap4_pm_info, cpu);unsigned int save_state =0 , cpu_logic_state =PWRDM_POWER_RET ;
@@ -268,6 +271,10 @@ int omap4_enter_lowpower(unsigned int cpcpu_clear_prev_logic_pwrst (cpu);pwrdm_set_next_pwrst (pm_info->pwrdm, power_state);pwrdm_set_logic_retst (pm_info->pwrdm, cpu_logic_state); + +if (rcuidle) +ct_cpuidle_enter (); +set_cpu_wakeup_addr (cpu,__pa_symbol (omap_pm_ops.resume)); omap_pm_ops.scu_prepare (cpu, power_state);l2x0_pwrst_prepare (cpu, save_state);
@@ -283,6 +290,9 @@ int omap4_enter_lowpower(unsigned int cpif (IS_PM44XX_ERRATUM (PM_OMAP4_ROM_SMP_BOOT_ERRATUM_GICD ) && cpu)gic_dist_enable (); +if (rcuidle) +ct_cpuidle_exit (); +/* * Restore the CPUx power state to ON otherwise CPUx * power domain can transitions to programmed low power
arch/arm/mach-omap2/pm44xx.c +1 -1
@@ -76,7 +76,7 @@ static int omap4_pm_suspend(void)* domain CSWR is not supported by hardware. * More details can be found in OMAP4430 TRM section 4.3.4.2. */ -omap4_enter_lowpower (cpu_id, cpu_suspend_state); +omap4_enter_lowpower (cpu_id, cpu_suspend_state, false);/* Restore next powerdomain state */ list_for_each_entry (pwrst, &pwrst_list, node) {
Peter Zijlstra
maintainerPATCH v3 39/51 arm,omap2: Use WFI for omap2_pm_idle()
In reply to Peter Zijlstraarch_cpu_idle() is a very simple idle interface and exposes only a single idle state and is expected to not require RCU and not do any tracing/instrumentation.
As such, omap2_pm_idle() is not a valid implementation. Replace it with a simple (shallow) omap2_do_wfi() call.
Omap2 doesn't have a cpuidle driver; but adding one would be the recourse to (re)gain the other idle states.
--- arch/arm/mach-omap2/pm24xx.c | 51 +------------------------------------------ 1 file changed, 2 insertions(+), 49 deletions(-)
arch/arm/mach-omap2/pm24xx.c +2 -49
@@ -116,50 +116,12 @@ static int omap2_enter_full_retention(vostatic int sti_console_enabled; -static int omap2_allow_mpu_retention (void ) -{ -if (!omap2xxx_cm_mpu_retention_allowed ()) -return 0 ; -if (sti_console_enabled) -return 0 ; - -return 1 ; -} - -static void omap2_enter_mpu_retention (void ) +static void omap2_do_wfi (void ) {const int zero =0 ; -/* The peripherals seem not to be able to wake up the MPU when -* it is in retention mode. */ -if (omap2_allow_mpu_retention ()) { -/* REVISIT: These write to reserved bits? */ -omap_prm_clear_mod_irqs (CORE_MOD ,PM_WKST1 , ~0 ); -omap_prm_clear_mod_irqs (CORE_MOD ,OMAP24XX_PM_WKST2 , ~0 ); -omap_prm_clear_mod_irqs (WKUP_MOD ,PM_WKST , ~0 ); - -/* Try to enter MPU retention */ -pwrdm_set_next_pwrst (mpu_pwrdm,PWRDM_POWER_RET ); - - }else { -/* Block MPU retention */ -pwrdm_set_next_pwrst (mpu_pwrdm,PWRDM_POWER_ON ); - } -/* WFI */ asm("mcr p15, 0, %0, c7, c0, 4" : :"r" (zero) :"memory" ,"cc" ); - -pwrdm_set_next_pwrst (mpu_pwrdm,PWRDM_POWER_ON ); -} - -static int omap2_can_sleep (void ) -{ -if (omap2xxx_cm_fclks_active ()) -return 0 ; -if (__clk_is_enabled (osc_ck)) -return 0 ; - -return 1 ; }static void omap2_pm_idle (void )
@@ -169,16 +131,7 @@ static void omap2_pm_idle(void)if (omap_irq_pending ())return ; - error =cpu_cluster_pm_enter (); -if (error || !omap2_can_sleep ()) { -omap2_enter_mpu_retention (); - goto out_cpu_cluster_pm; - } - -omap2_enter_full_retention (); - -out_cpu_cluster_pm: -cpu_cluster_pm_exit (); +omap2_do_wfi (); }static void __init prcm_setup_regs (void )
[tip: sched/core] arm, OMAP2: Use WFI for omap2_pm_idle()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 259c95afac6f2758760f158537e1d28358b4639c Gitweb: https://git.kernel.org/tip/259c95afac6f2758760f158537e1d28358b4639c Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:53 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
arm, OMAP2: Use WFI for omap2_pm_idle()
arch_cpu_idle() is a very simple idle interface and exposes only a single idle state and is expected to not require RCU and not do any tracing/instrumentation.
As such, omap2_pm_idle() is not a valid implementation. Replace it with a simple (shallow) omap2_do_wfi() call.
Omap2 doesn't have a cpuidle driver; but adding one would be the recourse to (re)gain the other idle states.
--- arch/arm/mach-omap2/pm24xx.c | 51 +---------------------------------- 1 file changed, 2 insertions(+), 49 deletions(-)
arch/arm/mach-omap2/pm24xx.c +2 -49
@@ -116,50 +116,12 @@ no_sleep:static int sti_console_enabled; -static int omap2_allow_mpu_retention (void ) -{ -if (!omap2xxx_cm_mpu_retention_allowed ()) -return 0 ; -if (sti_console_enabled) -return 0 ; - -return 1 ; -} - -static void omap2_enter_mpu_retention (void ) +static void omap2_do_wfi (void ) {const int zero =0 ; -/* The peripherals seem not to be able to wake up the MPU when -* it is in retention mode. */ -if (omap2_allow_mpu_retention ()) { -/* REVISIT: These write to reserved bits? */ -omap_prm_clear_mod_irqs (CORE_MOD ,PM_WKST1 , ~0 ); -omap_prm_clear_mod_irqs (CORE_MOD ,OMAP24XX_PM_WKST2 , ~0 ); -omap_prm_clear_mod_irqs (WKUP_MOD ,PM_WKST , ~0 ); - -/* Try to enter MPU retention */ -pwrdm_set_next_pwrst (mpu_pwrdm,PWRDM_POWER_RET ); - - }else { -/* Block MPU retention */ -pwrdm_set_next_pwrst (mpu_pwrdm,PWRDM_POWER_ON ); - } -/* WFI */ asm("mcr p15, 0, %0, c7, c0, 4" : :"r" (zero) :"memory" ,"cc" ); - -pwrdm_set_next_pwrst (mpu_pwrdm,PWRDM_POWER_ON ); -} - -static int omap2_can_sleep (void ) -{ -if (omap2xxx_cm_fclks_active ()) -return 0 ; -if (__clk_is_enabled (osc_ck)) -return 0 ; - -return 1 ; }static void omap2_pm_idle (void )
@@ -169,16 +131,7 @@ static void omap2_pm_idle(void)if (omap_irq_pending ())return ; - error =cpu_cluster_pm_enter (); -if (error || !omap2_can_sleep ()) { -omap2_enter_mpu_retention (); - goto out_cpu_cluster_pm; - } - -omap2_enter_full_retention (); - -out_cpu_cluster_pm: -cpu_cluster_pm_exit (); +omap2_do_wfi (); }static void __init prcm_setup_regs (void )
Peter Zijlstra
maintainerPATCH v3 40/51 cpuidle,powerdomain: Remove trace_.*_rcuidle()
In reply to Peter ZijlstraOMAP was the one and only user.
--- arch/arm/mach-omap2/powerdomain.c | 10 +++++----- drivers/base/power/runtime.c | 24 ++++++++++++------------ 2 files changed, 17 insertions(+), 17 deletions(-)
arch/arm/mach-omap2/powerdomain.c +5 -5
@@ -187,9 +187,9 @@ static int _pwrdm_state_switch(struct po trace_state = (PWRDM_TRACE_STATES_FLAG | ((next &OMAP_POWERSTATE_MASK ) <<8 ) | ((prev &OMAP_POWERSTATE_MASK ) <<0 )); -trace_power_domain_target_rcuidle (pwrdm->name, - trace_state, -raw_smp_processor_id ()); +trace_power_domain_target (pwrdm->name, + trace_state, +raw_smp_processor_id ()); }break ;default :
@@ -541,8 +541,8 @@ int pwrdm_set_next_pwrst(struct powerdomif (arch_pwrdm && arch_pwrdm->pwrdm_set_next_pwrst) {/* Trace the pwrdm desired target state */ -trace_power_domain_target_rcuidle (pwrdm->name, pwrst, -raw_smp_processor_id ()); +trace_power_domain_target (pwrdm->name, pwrst, +raw_smp_processor_id ());/* Program the pwrdm desired target state */ ret = arch_pwrdm->pwrdm_set_next_pwrst (pwrdm, pwrst); }
drivers/base/power/runtime.c +12 -12
@@ -442,7 +442,7 @@ static int rpm_idle(struct device *dev,int (*callback)(struct device *);int retval; -trace_rpm_idle_rcuidle (dev, rpmflags); +trace_rpm_idle (dev, rpmflags); retval =rpm_check_suspend_allowed (dev);if (retval <0 ) ;/* Conditions are wrong. */
@@ -481,7 +481,7 @@ static int rpm_idle(struct device *dev, dev->power.request_pending = true;queue_work (pm_wq, &dev->power.work); } -trace_rpm_return_int_rcuidle (dev, _THIS_IP_,0 ); +trace_rpm_return_int (dev, _THIS_IP_,0 );return 0 ; }
@@ -493,7 +493,7 @@ static int rpm_idle(struct device *dev,wake_up_all (&dev->power.wait_queue); out: -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval ? retval :rpm_suspend (dev, rpmflags |RPM_AUTO ); }
@@ -557,7 +557,7 @@ static int rpm_suspend(struct device *destruct device *parent =NULL ;int retval; -trace_rpm_suspend_rcuidle (dev, rpmflags); +trace_rpm_suspend (dev, rpmflags); repeat: retval =rpm_check_suspend_allowed (dev);
@@ -708,7 +708,7 @@ static int rpm_suspend(struct device *de } out: -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval;
@@ -760,7 +760,7 @@ static int rpm_resume(struct device *devstruct device *parent =NULL ;int retval =0 ; -trace_rpm_resume_rcuidle (dev, rpmflags); +trace_rpm_resume (dev, rpmflags); repeat:if (dev->power.runtime_error) {
@@ -925,7 +925,7 @@ static int rpm_resume(struct device *devspin_lock_irq (&dev->power.lock); } -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval; }
@@ -1081,7 +1081,7 @@ int __pm_runtime_idle(struct device *devif (retval <0 ) {return retval; }else if (retval >0 ) { -trace_rpm_usage_rcuidle (dev, rpmflags); +trace_rpm_usage (dev, rpmflags);return 0 ; } }
@@ -1119,7 +1119,7 @@ int __pm_runtime_suspend(struct device *if (retval <0 ) {return retval; }else if (retval >0 ) { -trace_rpm_usage_rcuidle (dev, rpmflags); +trace_rpm_usage (dev, rpmflags);return 0 ; } }
@@ -1202,7 +1202,7 @@ int pm_runtime_get_if_active(struct devi }else { retval =atomic_inc_not_zero (&dev->power.usage_count); } -trace_rpm_usage_rcuidle (dev,0 ); +trace_rpm_usage (dev,0 );spin_unlock_irqrestore (&dev->power.lock, flags);return retval;
@@ -1566,7 +1566,7 @@ void pm_runtime_allow(struct device *devif (ret ==0 )rpm_idle (dev,RPM_AUTO |RPM_ASYNC );else if (ret >0 ) -trace_rpm_usage_rcuidle (dev,RPM_AUTO |RPM_ASYNC ); +trace_rpm_usage (dev,RPM_AUTO |RPM_ASYNC ); out:spin_unlock_irq (&dev->power.lock);
@@ -1635,7 +1635,7 @@ static void update_autosuspend(struct deatomic_inc (&dev->power.usage_count);rpm_resume (dev,0 ); }else { -trace_rpm_usage_rcuidle (dev,0 ); +trace_rpm_usage (dev,0 ); } }
[tip: sched/core] cpuidle, ARM: OMAP2+: powerdomain: Remove trace_.*_rcuidle()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: db8f50861da6b1129b744da3998587a5cceeffeb Gitweb: https://git.kernel.org/tip/db8f50861da6b1129b744da3998587a5cceeffeb Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:54 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
cpuidle, ARM: OMAP2+: powerdomain: Remove trace_.*_rcuidle()
OMAP was the one and only user.
--- arch/arm/mach-omap2/powerdomain.c | 10 +++++----- drivers/base/power/runtime.c | 24 ++++++++++++------------ 2 files changed, 17 insertions(+), 17 deletions(-)
arch/arm/mach-omap2/powerdomain.c +5 -5
@@ -187,9 +187,9 @@ static int _pwrdm_state_switch(struct powerdomain *pwrdm, int flag) trace_state = (PWRDM_TRACE_STATES_FLAG | ((next &OMAP_POWERSTATE_MASK ) <<8 ) | ((prev &OMAP_POWERSTATE_MASK ) <<0 )); -trace_power_domain_target_rcuidle (pwrdm->name, - trace_state, -raw_smp_processor_id ()); +trace_power_domain_target (pwrdm->name, + trace_state, +raw_smp_processor_id ()); }break ;default :
@@ -541,8 +541,8 @@ int pwrdm_set_next_pwrst(struct powerdomain *pwrdm, u8 pwrst)if (arch_pwrdm && arch_pwrdm->pwrdm_set_next_pwrst) {/* Trace the pwrdm desired target state */ -trace_power_domain_target_rcuidle (pwrdm->name, pwrst, -raw_smp_processor_id ()); +trace_power_domain_target (pwrdm->name, pwrst, +raw_smp_processor_id ());/* Program the pwrdm desired target state */ ret = arch_pwrdm->pwrdm_set_next_pwrst (pwrdm, pwrst); }
drivers/base/power/runtime.c +12 -12
@@ -468,7 +468,7 @@ static int rpm_idle(struct device *dev, int rpmflags)int (*callback)(struct device *);int retval; -trace_rpm_idle_rcuidle (dev, rpmflags); +trace_rpm_idle (dev, rpmflags); retval =rpm_check_suspend_allowed (dev);if (retval <0 ) ;/* Conditions are wrong. */
@@ -508,7 +508,7 @@ static int rpm_idle(struct device *dev, int rpmflags) dev->power.request_pending = true;queue_work (pm_wq, &dev->power.work); } -trace_rpm_return_int_rcuidle (dev, _THIS_IP_,0 ); +trace_rpm_return_int (dev, _THIS_IP_,0 );return 0 ; }
@@ -530,7 +530,7 @@ static int rpm_idle(struct device *dev, int rpmflags)wake_up_all (&dev->power.wait_queue); out: -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval ? retval :rpm_suspend (dev, rpmflags |RPM_AUTO ); }
@@ -562,7 +562,7 @@ static int rpm_suspend(struct device *dev, int rpmflags)struct device *parent =NULL ;int retval; -trace_rpm_suspend_rcuidle (dev, rpmflags); +trace_rpm_suspend (dev, rpmflags); repeat: retval =rpm_check_suspend_allowed (dev);
@@ -713,7 +713,7 @@ static int rpm_suspend(struct device *dev, int rpmflags) } out: -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval;
@@ -765,7 +765,7 @@ static int rpm_resume(struct device *dev, int rpmflags)struct device *parent =NULL ;int retval =0 ; -trace_rpm_resume_rcuidle (dev, rpmflags); +trace_rpm_resume (dev, rpmflags); repeat:if (dev->power.runtime_error) {
@@ -935,7 +935,7 @@ static int rpm_resume(struct device *dev, int rpmflags)spin_lock_irq (&dev->power.lock); } -trace_rpm_return_int_rcuidle (dev, _THIS_IP_, retval); +trace_rpm_return_int (dev, _THIS_IP_, retval);return retval; }
@@ -1091,7 +1091,7 @@ int __pm_runtime_idle(struct device *dev, int rpmflags)if (retval <0 ) {return retval; }else if (retval >0 ) { -trace_rpm_usage_rcuidle (dev, rpmflags); +trace_rpm_usage (dev, rpmflags);return 0 ; } }
@@ -1129,7 +1129,7 @@ int __pm_runtime_suspend(struct device *dev, int rpmflags)if (retval <0 ) {return retval; }else if (retval >0 ) { -trace_rpm_usage_rcuidle (dev, rpmflags); +trace_rpm_usage (dev, rpmflags);return 0 ; } }
@@ -1212,7 +1212,7 @@ int pm_runtime_get_if_active(struct device *dev, bool ign_usage_count) }else { retval =atomic_inc_not_zero (&dev->power.usage_count); } -trace_rpm_usage_rcuidle (dev,0 ); +trace_rpm_usage (dev,0 );spin_unlock_irqrestore (&dev->power.lock, flags);return retval;
@@ -1576,7 +1576,7 @@ void pm_runtime_allow(struct device *dev)if (ret ==0 )rpm_idle (dev,RPM_AUTO |RPM_ASYNC );else if (ret >0 ) -trace_rpm_usage_rcuidle (dev,RPM_AUTO |RPM_ASYNC ); +trace_rpm_usage (dev,RPM_AUTO |RPM_ASYNC ); out:spin_unlock_irq (&dev->power.lock);
@@ -1646,7 +1646,7 @@ static void update_autosuspend(struct device *dev, int old_delay, int old_use)atomic_inc (&dev->power.usage_count);rpm_resume (dev,0 ); }else { -trace_rpm_usage_rcuidle (dev,0 ); +trace_rpm_usage (dev,0 ); } }
Peter Zijlstra
maintainerPATCH v3 41/51 cpuidle,clk: Remove trace_.*_rcuidle()
In reply to Peter ZijlstraOMAP was the one and only user.
--- drivers/clk/clk.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/clk/clk.c +4 -4
@@ -978,12 +978,12 @@ static void clk_core_disable(struct clk_if (--core->enable_count >0 )return ; -trace_clk_disable_rcuidle (core); +trace_clk_disable (core);if (core->ops->disable) core->ops->disable (core->hw); -trace_clk_disable_complete_rcuidle (core); +trace_clk_disable_complete (core);clk_core_disable (core->parent); }
@@ -1037,12 +1037,12 @@ static int clk_core_enable(struct clk_coif (ret)return ret; -trace_clk_enable_rcuidle (core); +trace_clk_enable (core);if (core->ops->enable) ret = core->ops->enable (core->hw); -trace_clk_enable_complete_rcuidle (core); +trace_clk_enable_complete (core);if (ret) {clk_core_disable (core->parent);
Re: [PATCH v3 41/51] cpuidle,clk: Remove trace_.*_rcuidle()
Quoting Peter Zijlstra (2023-01-12 11:43:55)
Quoted 9 lines
OMAP was the one and only user.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Reviewed-by Ulf Hansson <[email protected]>Tested-by Tony Lindgren <[email protected]>Tested-by Ulf Hansson <[email protected]>---
[tip: sched/core] cpuidle, clk: Remove trace_.*_rcuidle()
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: ca502fc6d9d90c8384eb3e97e964177521d9df48 Gitweb: https://git.kernel.org/tip/ca502fc6d9d90c8384eb3e97e964177521d9df48 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:55 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
cpuidle, clk: Remove trace_.*_rcuidle()
OMAP was the one and only user.
--- drivers/clk/clk.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/clk/clk.c +4 -4
@@ -1055,12 +1055,12 @@ static void clk_core_disable(struct clk_core *core)if (--core->enable_count >0 )return ; -trace_clk_disable_rcuidle (core); +trace_clk_disable (core);if (core->ops->disable) core->ops->disable (core->hw); -trace_clk_disable_complete_rcuidle (core); +trace_clk_disable_complete (core);clk_core_disable (core->parent); }
@@ -1114,12 +1114,12 @@ static int clk_core_enable(struct clk_core *core)if (ret)return ret; -trace_clk_enable_rcuidle (core); +trace_clk_enable (core);if (core->ops->enable) ret = core->ops->enable (core->hw); -trace_clk_enable_complete_rcuidle (core); +trace_clk_enable_complete (core);if (ret) {clk_core_disable (core->parent);
Peter Zijlstra
maintainerPATCH v3 42/51 ubsan: Fix objtool UACCESS warns
In reply to Peter Zijlstraclang-14 allyesconfig gives:
vmlinux.o: warning: objtool: emulator_cmpxchg_emulated+0x705: call to __ubsan_handle_load_invalid_value() with UACCESS enabled vmlinux.o: warning: objtool: paging64_update_accessed_dirty_bits+0x39e: call to __ubsan_handle_load_invalid_value() with UACCESS enabled vmlinux.o: warning: objtool: paging32_update_accessed_dirty_bits+0x390: call to __ubsan_handle_load_invalid_value() with UACCESS enabled vmlinux.o: warning: objtool: ept_update_accessed_dirty_bits+0x43f: call to __ubsan_handle_load_invalid_value() with UACCESS enabled
Add the required eflags save/restore and whitelist the thing.
--- lib/ubsan.c | 5 ++++- tools/objtool/check.c | 1 + 2 files changed, 5 insertions(+), 1 deletion(-)
lib/ubsan.c +4 -1
@@ -340,9 +340,10 @@ void __ubsan_handle_load_invalid_value(v {struct invalid_value_data *data = _data;char val_str[VALUE_LENGTH ]; +unsigned long ua_flags =user_access_save ();if (suppress_report (&data->location)) -return ; + goto out;ubsan_prologue (&data->location,"invalid-load" );
@@ -352,6 +353,8 @@ void __ubsan_handle_load_invalid_value(v val_str, data->type->type_name);ubsan_epilogue (); +out: +user_access_restore (ua_flags); }EXPORT_SYMBOL (__ubsan_handle_load_invalid_value);
tools/objtool/check.c +1 -0
@@ -1068,6 +1068,7 @@ static const char *uaccess_safe_builtin["__ubsan_handle_type_mismatch" ,"__ubsan_handle_type_mismatch_v1" ,"__ubsan_handle_shift_out_of_bounds" , +"__ubsan_handle_load_invalid_value" ,/* misc */ "csum_partial_copy_generic" , "copy_mc_fragile",
[tip: sched/core] ubsan: Fix objtool UACCESS warns
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: f18b0d7ee815abb33eb1cc540fdde64fbb922d1a Gitweb: https://git.kernel.org/tip/f18b0d7ee815abb33eb1cc540fdde64fbb922d1a Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:56 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
ubsan: Fix objtool UACCESS warns
clang-14 allyesconfig gives:
vmlinux.o: warning: objtool: emulator_cmpxchg_emulated+0x705: call to __ubsan_handle_load_invalid_value() with UACCESS enabled vmlinux.o: warning: objtool: paging64_update_accessed_dirty_bits+0x39e: call to __ubsan_handle_load_invalid_value() with UACCESS enabled vmlinux.o: warning: objtool: paging32_update_accessed_dirty_bits+0x390: call to __ubsan_handle_load_invalid_value() with UACCESS enabled vmlinux.o: warning: objtool: ept_update_accessed_dirty_bits+0x43f: call to __ubsan_handle_load_invalid_value() with UACCESS enabled
Add the required eflags save/restore and whitelist the thing.
--- lib/ubsan.c | 5 ++++- tools/objtool/check.c | 1 + 2 files changed, 5 insertions(+), 1 deletion(-)
lib/ubsan.c +4 -1
@@ -339,9 +339,10 @@ void __ubsan_handle_load_invalid_value(void *_data, void *val) {struct invalid_value_data *data = _data;char val_str[VALUE_LENGTH ]; +unsigned long ua_flags =user_access_save ();if (suppress_report (&data->location)) -return ; + goto out;ubsan_prologue (&data->location,"invalid-load" );
@@ -351,6 +352,8 @@ void __ubsan_handle_load_invalid_value(void *_data, void *val) val_str, data->type->type_name);ubsan_epilogue (); +out: +user_access_restore (ua_flags); }EXPORT_SYMBOL (__ubsan_handle_load_invalid_value);
tools/objtool/check.c +1 -0
@@ -1216,6 +1216,7 @@ static const char *uaccess_safe_builtin[] = {"__ubsan_handle_type_mismatch" ,"__ubsan_handle_type_mismatch_v1" ,"__ubsan_handle_shift_out_of_bounds" , +"__ubsan_handle_load_invalid_value" ,/* misc */ "csum_partial_copy_generic" , "copy_mc_fragile",
Peter Zijlstra
maintainerPATCH v3 43/51 intel_idle: Add force_irq_on module param
In reply to Peter ZijlstraFor testing purposes.
--- drivers/idle/intel_idle.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/idle/intel_idle.c +6 -1
@@ -1787,6 +1787,9 @@ static bool __init intel_idle_verify_cstreturn true; } +static bool force_irq_on__read_mostly ; +module_param (force_irq_on, bool,0444 ); +static void __init intel_idle_init_cstates_icpu (struct cpuidle_driver *drv) {int cstate;
@@ -1838,8 +1841,10 @@ static void __init intel_idle_init_cstat/* Structure copy. */ drv->states[drv->state_count] = cpuidle_state_table[cstate]; -if (cpuidle_state_table[cstate].flags &CPUIDLE_FLAG_IRQ_ENABLE ) +if ((cpuidle_state_table[cstate].flags &CPUIDLE_FLAG_IRQ_ENABLE ) || force_irq_on) { +printk ("intel_idle: forced intel_idle_irq for state %d\n" , cstate); drv->states[drv->state_count].enter = intel_idle_irq; + }if (cpu_feature_enabled (X86_FEATURE_KERNEL_IBRS ) && cpuidle_state_table[cstate].flags &CPUIDLE_FLAG_IBRS ) {
[tip: sched/core] intel_idle: Add force_irq_on module param
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 365bd03ff63fb7433a49c453472cf54830b677bf Gitweb: https://git.kernel.org/tip/365bd03ff63fb7433a49c453472cf54830b677bf Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:57 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
intel_idle: Add force_irq_on module param
For testing purposes.
--- drivers/idle/intel_idle.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/idle/intel_idle.c +6 -1
@@ -1837,6 +1837,9 @@ static bool __init intel_idle_verify_cstate(unsigned int mwait_hint)return true; } +static bool force_irq_on__read_mostly ; +module_param (force_irq_on, bool,0444 ); +static void __init intel_idle_init_cstates_icpu (struct cpuidle_driver *drv) {int cstate;
@@ -1889,8 +1892,10 @@ static void __init intel_idle_init_cstates_icpu(struct cpuidle_driver *drv)/* Structure copy. */ drv->states[drv->state_count] = cpuidle_state_table[cstate]; -if (cpuidle_state_table[cstate].flags &CPUIDLE_FLAG_IRQ_ENABLE ) +if ((cpuidle_state_table[cstate].flags &CPUIDLE_FLAG_IRQ_ENABLE ) || force_irq_on) { +printk ("intel_idle: forced intel_idle_irq for state %d\n" , cstate); drv->states[drv->state_count].enter = intel_idle_irq; + }if (cpu_feature_enabled (X86_FEATURE_KERNEL_IBRS ) && cpuidle_state_table[cstate].flags &CPUIDLE_FLAG_IBRS ) {
Peter Zijlstra
maintainerPATCH v3 44/51 entry,kasan,x86: Disallow overriding mem*() functions
In reply to Peter ZijlstraKASAN cannot just hijack the mem*() functions, it needs to emit __asan_mem*() variants if it wants instrumentation (other sanitizers already do this).
vmlinux.o: warning: objtool: sync_regs+0x24: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: vc_switch_off_ist+0xbe: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: fixup_bad_iret+0x36: call to memset() leaves .noinstr.text section vmlinux.o: warning: objtool: __sev_get_ghcb+0xa0: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: __sev_put_ghcb+0x35: call to memcpy() leaves .noinstr.text section
Remove the weak aliases to ensure nobody hijacks these functions and add them to the noinstr section.
--- arch/x86/lib/memcpy_64.S | 5 ++--- arch/x86/lib/memmove_64.S | 4 +++- arch/x86/lib/memset_64.S | 4 +++- mm/kasan/kasan.h | 4 ++++ mm/kasan/shadow.c | 38 ++++++++++++++++++++++++++++++++++++++ tools/objtool/check.c | 3 +++ 6 files changed, 53 insertions(+), 5 deletions(-)
arch/x86/lib/memcpy_64.S +2 -3
@@ -7,7 +7,7 @@ #include <asm/alternative.h> #include <asm/export.h> -.pushsection .noinstr.text, "ax" +.section .noinstr.text, "ax" /* * We build a jump to memcpy_orig by default which gets NOPped out on
@@ -42,7 +42,7 @@ SYM_FUNC_START(__memcpy) SYM_FUNC_END(__memcpy) EXPORT_SYMBOL(__memcpy) -SYM_FUNC_ALIAS_WEAK(memcpy, __memcpy) +SYM_FUNC_ALIAS(memcpy, __memcpy) EXPORT_SYMBOL(memcpy) /*
@@ -183,4 +183,3 @@ SYM_FUNC_START_LOCAL(memcpy_orig) RET SYM_FUNC_END(memcpy_orig) -.popsection
arch/x86/lib/memmove_64.S +3 -1
@@ -13,6 +13,8 @@ #undef memmove +.section .noinstr.text, "ax" + /* * Implement memmove(). This can handle overlap between src and dst. *
@@ -213,5 +215,5 @@ SYM_FUNC_START(__memmove) SYM_FUNC_END(__memmove) EXPORT_SYMBOL(__memmove) -SYM_FUNC_ALIAS_WEAK(memmove, __memmove) +SYM_FUNC_ALIAS(memmove, __memmove) EXPORT_SYMBOL(memmove)
arch/x86/lib/memset_64.S +3 -1
@@ -6,6 +6,8 @@ #include <asm/alternative.h> #include <asm/export.h> +.section .noinstr.text, "ax" + /* * ISO C memset - set a memory block to a byte value. This function uses fast * string to get better performance than the original function. The code is
@@ -43,7 +45,7 @@ SYM_FUNC_START(__memset) SYM_FUNC_END(__memset) EXPORT_SYMBOL(__memset) -SYM_FUNC_ALIAS_WEAK(memset, __memset) +SYM_FUNC_ALIAS(memset, __memset) EXPORT_SYMBOL(memset) /*
mm/kasan/kasan.h +4 -0
@@ -551,6 +551,10 @@ void __asan_set_shadow_f3(const void *advoid __asan_set_shadow_f5 (const void *addr,size_t size);void __asan_set_shadow_f8 (const void *addr,size_t size); +void *__asan_memset (void *addr,int c,size_t len); +void *__asan_memmove (void *dest,const void *src,size_t len); +void *__asan_memcpy (void *dest,const void *src,size_t len); +void __hwasan_load1_noabort (unsigned long addr);void __hwasan_store1_noabort (unsigned long addr);void __hwasan_load2_noabort (unsigned long addr);
mm/kasan/shadow.c +38 -0
@@ -38,6 +38,12 @@ bool __kasan_check_write(const volatile }EXPORT_SYMBOL (__kasan_check_write); +#ifndef CONFIG_GENERIC_ENTRY +/* +* CONFIG_GENERIC_ENTRY relies on compiler emitted mem*() calls to not be +* instrumented. KASAN enabled toolchains should emit __asan_mem*() functions +* for the sites they want to instrument. +*/ #undef memsetvoid *memset (void *addr,int c,size_t len) {
@@ -68,6 +74,38 @@ void *memcpy(void *dest, const void *srcreturn __memcpy (dest, src, len); } +#endif + +void *__asan_memset (void *addr,int c,size_t len) +{ +if (!kasan_check_range ((unsigned long )addr, len, true, _RET_IP_)) +return NULL ; + +return __memset (addr, c, len); +} +EXPORT_SYMBOL (__asan_memset); + +#ifdef __HAVE_ARCH_MEMMOVE +void *__asan_memmove (void *dest,const void *src,size_t len) +{ +if (!kasan_check_range ((unsigned long )src, len, false, _RET_IP_) || + !kasan_check_range ((unsigned long )dest, len, true, _RET_IP_)) +return NULL ; + +return __memmove (dest, src, len); +} +EXPORT_SYMBOL (__asan_memmove); +#endif + +void *__asan_memcpy (void *dest,const void *src,size_t len) +{ +if (!kasan_check_range ((unsigned long )src, len, false, _RET_IP_) || + !kasan_check_range ((unsigned long )dest, len, true, _RET_IP_)) +return NULL ; + +return __memcpy (dest, src, len); +} +EXPORT_SYMBOL (__asan_memcpy);void kasan_poison (const void *addr,size_t size,u8 value,bool init) {
tools/objtool/check.c +3 -0
@@ -956,6 +956,9 @@ static const char *uaccess_safe_builtin["__asan_store16_noabort" ,"__kasan_check_read" ,"__kasan_check_write" , +"__asan_memset" , +"__asan_memmove" , +"__asan_memcpy" ,/* KASAN in-line */ "__asan_report_load_n_noabort" , "__asan_report_load1_noabort",
[tip: sched/core] entry, kasan, x86: Disallow overriding mem*() functions
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 69d4c0d3218692ffa56b0e1b9c76c50c699d7044 Gitweb: https://git.kernel.org/tip/69d4c0d3218692ffa56b0e1b9c76c50c699d7044 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:58 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
entry, kasan, x86: Disallow overriding mem*() functions
KASAN cannot just hijack the mem*() functions, it needs to emit __asan_mem*() variants if it wants instrumentation (other sanitizers already do this).
vmlinux.o: warning: objtool: sync_regs+0x24: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: vc_switch_off_ist+0xbe: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: fixup_bad_iret+0x36: call to memset() leaves .noinstr.text section vmlinux.o: warning: objtool: __sev_get_ghcb+0xa0: call to memcpy() leaves .noinstr.text section vmlinux.o: warning: objtool: __sev_put_ghcb+0x35: call to memcpy() leaves .noinstr.text section
Remove the weak aliases to ensure nobody hijacks these functions and add them to the noinstr section.
--- arch/x86/lib/memcpy_64.S | 5 ++--- arch/x86/lib/memmove_64.S | 4 +++- arch/x86/lib/memset_64.S | 4 +++- mm/kasan/kasan.h | 4 ++++- mm/kasan/shadow.c | 38 ++++++++++++++++++++++++++++++++++++++- tools/objtool/check.c | 3 +++- 6 files changed, 53 insertions(+), 5 deletions(-)
arch/x86/lib/memcpy_64.S +2 -3
@@ -8,7 +8,7 @@ #include <asm/alternative.h> #include <asm/export.h> -.pushsection .noinstr.text, "ax" +.section .noinstr.text, "ax" /* * We build a jump to memcpy_orig by default which gets NOPped out on
@@ -43,7 +43,7 @@ SYM_TYPED_FUNC_START(__memcpy) SYM_FUNC_END(__memcpy) EXPORT_SYMBOL(__memcpy) -SYM_FUNC_ALIAS_WEAK(memcpy, __memcpy) +SYM_FUNC_ALIAS(memcpy, __memcpy) EXPORT_SYMBOL(memcpy) /*
@@ -184,4 +184,3 @@ SYM_FUNC_START_LOCAL(memcpy_orig) RET SYM_FUNC_END(memcpy_orig) -.popsection
arch/x86/lib/memmove_64.S +3 -1
@@ -13,6 +13,8 @@ #undef memmove +.section .noinstr.text, "ax" + /* * Implement memmove(). This can handle overlap between src and dst. *
@@ -213,5 +215,5 @@ SYM_FUNC_START(__memmove) SYM_FUNC_END(__memmove) EXPORT_SYMBOL(__memmove) -SYM_FUNC_ALIAS_WEAK(memmove, __memmove) +SYM_FUNC_ALIAS(memmove, __memmove) EXPORT_SYMBOL(memmove)
arch/x86/lib/memset_64.S +3 -1
@@ -6,6 +6,8 @@ #include <asm/alternative.h> #include <asm/export.h> +.section .noinstr.text, "ax" + /* * ISO C memset - set a memory block to a byte value. This function uses fast * string to get better performance than the original function. The code is
@@ -43,7 +45,7 @@ SYM_FUNC_START(__memset) SYM_FUNC_END(__memset) EXPORT_SYMBOL(__memset) -SYM_FUNC_ALIAS_WEAK(memset, __memset) +SYM_FUNC_ALIAS(memset, __memset) EXPORT_SYMBOL(memset) /*
mm/kasan/kasan.h +4 -0
@@ -618,6 +618,10 @@ void __asan_set_shadow_f3(const void *addr, size_t size);void __asan_set_shadow_f5 (const void *addr,size_t size);void __asan_set_shadow_f8 (const void *addr,size_t size); +void *__asan_memset (void *addr,int c,size_t len); +void *__asan_memmove (void *dest,const void *src,size_t len); +void *__asan_memcpy (void *dest,const void *src,size_t len); +void __hwasan_load1_noabort (unsigned long addr);void __hwasan_store1_noabort (unsigned long addr);void __hwasan_load2_noabort (unsigned long addr);
mm/kasan/shadow.c +38 -0
@@ -38,6 +38,12 @@ bool __kasan_check_write(const volatile void *p, unsigned int size) }EXPORT_SYMBOL (__kasan_check_write); +#ifndef CONFIG_GENERIC_ENTRY +/* +* CONFIG_GENERIC_ENTRY relies on compiler emitted mem*() calls to not be +* instrumented. KASAN enabled toolchains should emit __asan_mem*() functions +* for the sites they want to instrument. +*/ #undef memsetvoid *memset (void *addr,int c,size_t len) {
@@ -68,6 +74,38 @@ void *memcpy(void *dest, const void *src, size_t len)return __memcpy (dest, src, len); } +#endif + +void *__asan_memset (void *addr,int c,size_t len) +{ +if (!kasan_check_range ((unsigned long )addr, len, true, _RET_IP_)) +return NULL ; + +return __memset (addr, c, len); +} +EXPORT_SYMBOL (__asan_memset); + +#ifdef __HAVE_ARCH_MEMMOVE +void *__asan_memmove (void *dest,const void *src,size_t len) +{ +if (!kasan_check_range ((unsigned long )src, len, false, _RET_IP_) || + !kasan_check_range ((unsigned long )dest, len, true, _RET_IP_)) +return NULL ; + +return __memmove (dest, src, len); +} +EXPORT_SYMBOL (__asan_memmove); +#endif + +void *__asan_memcpy (void *dest,const void *src,size_t len) +{ +if (!kasan_check_range ((unsigned long )src, len, false, _RET_IP_) || + !kasan_check_range ((unsigned long )dest, len, true, _RET_IP_)) +return NULL ; + +return __memcpy (dest, src, len); +} +EXPORT_SYMBOL (__asan_memcpy);void kasan_poison (const void *addr,size_t size,u8 value,bool init) {
tools/objtool/check.c +3 -0
@@ -1074,6 +1074,9 @@ static const char *uaccess_safe_builtin[] = {"__asan_store16_noabort" ,"__kasan_check_read" ,"__kasan_check_write" , +"__asan_memset" , +"__asan_memmove" , +"__asan_memcpy" ,/* KASAN in-line */ "__asan_report_load_n_noabort" , "__asan_report_load1_noabort",
Peter Zijlstra
maintainerPATCH v3 45/51 sched: Always inline __this_cpu_preempt_check()
In reply to Peter Zijlstravmlinux.o: warning: objtool: in_entry_stack+0x9: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: default_do_nmi+0x10: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: fpu_idle_fpregs+0x41: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: kvm_read_and_reset_apf_flags+0x1: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: lockdep_hardirqs_on+0xb0: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: lockdep_hardirqs_off+0xae: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_nmi_enter+0x69: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_nmi_exit+0x32: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0x9: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_idle_enter+0x43: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_idle_enter_s2idle+0x45: call to __this_cpu_preempt_check() leaves .noinstr.text section
--- include/linux/percpu-defs.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/percpu-defs.h +1 -1
@@ -310,7 +310,7 @@ extern void __bad_size_call_parameter(vo#ifdef CONFIG_DEBUG_PREEMPT extern void __this_cpu_preempt_check (const char *op);#else -static inline void __this_cpu_preempt_check (const char *op) { } +static __always_inline void __this_cpu_preempt_check(const char *op) { }#endif #define __pcpu_size_call_return (stem, variable) \
[tip: sched/core] sched/core: Always inline __this_cpu_preempt_check()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: f176d4ccb30747231831f66779697afa6d738b3c Gitweb: https://git.kernel.org/tip/f176d4ccb30747231831f66779697afa6d738b3c Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:43:59 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
sched/core: Always inline __this_cpu_preempt_check()
Quite a few unnecessary instrumentation calls are generated via the no-op __this_cpu_preempt_check() call, if it gets uninlined by the compiler:
vmlinux.o: warning: objtool: in_entry_stack+0x9: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: default_do_nmi+0x10: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: fpu_idle_fpregs+0x41: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: kvm_read_and_reset_apf_flags+0x1: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: lockdep_hardirqs_on+0xb0: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: lockdep_hardirqs_off+0xae: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_nmi_enter+0x69: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: irqentry_nmi_exit+0x32: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0x9: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_idle_enter+0x43: call to __this_cpu_preempt_check() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_idle_enter_s2idle+0x45: call to __this_cpu_preempt_check() leaves .noinstr.text section
Mark it __always_inline.
--- include/linux/percpu-defs.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/percpu-defs.h +1 -1
@@ -310,7 +310,7 @@ extern void __bad_size_call_parameter(void);#ifdef CONFIG_DEBUG_PREEMPT extern void __this_cpu_preempt_check (const char *op);#else -static inline void __this_cpu_preempt_check (const char *op) { } +static __always_inline void __this_cpu_preempt_check(const char *op) { }#endif #define __pcpu_size_call_return (stem, variable) \
Peter Zijlstra
maintainerPATCH v3 46/51 arm64,riscv,perf: Remove RCU_NONIDLE() usage
In reply to Peter ZijlstraThe PM notifiers should no longer be ran with RCU disabled (per the previous patches), as such this hack is no longer required either.
--- drivers/perf/arm_pmu.c | 11 +---------- drivers/perf/riscv_pmu_sbi.c | 8 +------- 2 files changed, 2 insertions(+), 17 deletions(-)
drivers/perf/arm_pmu.c +1 -10
@@ -762,17 +762,8 @@ static void cpu_pm_pmu_setup(struct arm_case CPU_PM_ENTER_FAILED :/* * Restore and enable the counter. -* armpmu_start() indirectly calls -* -* perf_event_update_userpage() -* -* that requires RCU read locking to be functional, -* wrap the call within RCU_NONIDLE to make the -* RCU subsystem aware this cpu is not idle from -* an RCU perspective for the armpmu_start() call -* duration. */ -RCU_NONIDLE (armpmu_start (event,PERF_EF_RELOAD )); +armpmu_start (event,PERF_EF_RELOAD );break ;default :break ;
drivers/perf/riscv_pmu_sbi.c +1 -7
@@ -747,14 +747,8 @@ static int riscv_pm_pmu_notify(struct nocase CPU_PM_ENTER_FAILED :/* * Restore and enable the counter. -* -* Requires RCU read locking to be functional, -* wrap the call within RCU_NONIDLE to make the -* RCU subsystem aware this cpu is not idle from -* an RCU perspective for the riscv_pmu_start() call -* duration. */ -RCU_NONIDLE (riscv_pmu_start (event,PERF_EF_RELOAD )); +riscv_pmu_start (event,PERF_EF_RELOAD );break ;default :break ;
[tip: sched/core] arm64, riscv, perf: Remove RCU_NONIDLE() usage
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 1c38b0615f84f622cd6c821aa40be60fc7f7a096 Gitweb: https://git.kernel.org/tip/1c38b0615f84f622cd6c821aa40be60fc7f7a096 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:44:00 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
arm64, riscv, perf: Remove RCU_NONIDLE() usage
The PM notifiers should no longer be ran with RCU disabled (per the previous patches), as such this hack is no longer required either.
--- drivers/perf/arm_pmu.c | 11 +---------- drivers/perf/riscv_pmu_sbi.c | 8 +------- 2 files changed, 2 insertions(+), 17 deletions(-)
drivers/perf/arm_pmu.c +1 -10
@@ -758,17 +758,8 @@ static void cpu_pm_pmu_setup(struct arm_pmu *armpmu, unsigned long cmd)case CPU_PM_ENTER_FAILED :/* * Restore and enable the counter. -* armpmu_start() indirectly calls -* -* perf_event_update_userpage() -* -* that requires RCU read locking to be functional, -* wrap the call within RCU_NONIDLE to make the -* RCU subsystem aware this cpu is not idle from -* an RCU perspective for the armpmu_start() call -* duration. */ -RCU_NONIDLE (armpmu_start (event,PERF_EF_RELOAD )); +armpmu_start (event,PERF_EF_RELOAD );break ;default :break ;
drivers/perf/riscv_pmu_sbi.c +1 -7
@@ -771,14 +771,8 @@ static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd,case CPU_PM_ENTER_FAILED :/* * Restore and enable the counter. -* -* Requires RCU read locking to be functional, -* wrap the call within RCU_NONIDLE to make the -* RCU subsystem aware this cpu is not idle from -* an RCU perspective for the riscv_pmu_start() call -* duration. */ -RCU_NONIDLE (riscv_pmu_start (event,PERF_EF_RELOAD )); +riscv_pmu_start (event,PERF_EF_RELOAD );break ;default :break ;
Peter Zijlstra
maintainerPATCH v3 47/51 cpuidle: Ensure ct_cpuidle_enter() is always called from noinstr/__cpuidle
In reply to Peter ZijlstraTracing (kprobes included) and other compiler instrumentation relies on a normal kernel runtime. Therefore all functions that disable RCU should be noinstr, as should all functions that are called while RCU is disabled.
--- drivers/cpuidle/cpuidle.c | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-)
drivers/cpuidle/cpuidle.c +28 -9
@@ -137,11 +137,13 @@ int cpuidle_find_deepest_state(struct cp }#ifdef CONFIG_SUSPEND -static void enter_s2idle_proper (struct cpuidle_driver *drv, -struct cpuidle_device *dev,int index) +static noinstr void enter_s2idle_proper(struct cpuidle_driver *drv, +struct cpuidle_device *dev,int index) { -ktime_t time_start, time_end;struct cpuidle_state *target_state = &drv->states[index]; +ktime_t time_start, time_end; + +instrumentation_begin (); time_start =ns_to_ktime (local_clock ());
@@ -152,13 +154,18 @@ static void enter_s2idle_proper(struct c* suspended is generally unsafe. */ stop_critical_timings (); -if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) {ct_cpuidle_enter (); +/* Annotate away the indirect call */ +instrumentation_begin (); + } target_state->enter_s2idle (dev, drv, index);if (WARN_ON_ONCE (!irqs_disabled ()))raw_local_irq_disable (); -if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) { +instrumentation_end ();ct_cpuidle_exit (); + }tick_unfreeze ();start_critical_timings ();
@@ -166,6 +173,7 @@ static void enter_s2idle_proper(struct c dev->states_usage[index].s2idle_time +=ktime_us_delta (time_end, time_start); dev->states_usage[index].s2idle_usage++; +instrumentation_end (); }/**
@@ -200,8 +208,9 @@ int cpuidle_enter_s2idle(struct cpuidle_* @drv: cpuidle driver for this cpu * @index: index into the states table in @drv of the state to enter */ -int cpuidle_enter_state (struct cpuidle_device *dev,struct cpuidle_driver *drv, -int index) +noinstr int cpuidle_enter_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv, +int index) {int entered_state;
@@ -209,6 +218,8 @@ int cpuidle_enter_state(struct cpuidle_dbool broadcast = !!(target_state->flags &CPUIDLE_FLAG_TIMER_STOP );ktime_t time_start, time_end; +instrumentation_begin (); +/* * Tell the time framework to switch to a broadcast timer because our * local timer will be shut down. If a local timer is used from another
@@ -235,15 +246,21 @@ int cpuidle_enter_state(struct cpuidle_d time_start =ns_to_ktime (local_clock ());stop_critical_timings (); -if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) {ct_cpuidle_enter (); +/* Annotate away the indirect call */ +instrumentation_begin (); + } entered_state = target_state->enter (dev, drv, index); +if (WARN_ONCE (!irqs_disabled (),"%ps leaked IRQ state" , target_state->enter))raw_local_irq_disable (); -if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) { +instrumentation_end ();ct_cpuidle_exit (); + }start_critical_timings ();sched_clock_idle_wakeup_event ();
@@ -306,6 +323,8 @@ int cpuidle_enter_state(struct cpuidle_d dev->states_usage[index].rejected++; } +instrumentation_end (); +return entered_state; }
[tip: sched/core] cpuidle: Ensure ct_cpuidle_enter() is always called from noinstr/__cpuidle
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 17cc2b5525856d7374f09b28ba1faf1fa61b2352 Gitweb: https://git.kernel.org/tip/17cc2b5525856d7374f09b28ba1faf1fa61b2352 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:44:01 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
cpuidle: Ensure ct_cpuidle_enter() is always called from noinstr/__cpuidle
Tracing (kprobes included) and other compiler instrumentation relies on a normal kernel runtime. Therefore all functions that disable RCU should be noinstr, as should all functions that are called while RCU is disabled.
--- drivers/cpuidle/cpuidle.c | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-)
drivers/cpuidle/cpuidle.c +28 -9
@@ -137,11 +137,13 @@ int cpuidle_find_deepest_state(struct cpuidle_driver *drv, }#ifdef CONFIG_SUSPEND -static void enter_s2idle_proper (struct cpuidle_driver *drv, -struct cpuidle_device *dev,int index) +static noinstr void enter_s2idle_proper(struct cpuidle_driver *drv, +struct cpuidle_device *dev,int index) { -ktime_t time_start, time_end;struct cpuidle_state *target_state = &drv->states[index]; +ktime_t time_start, time_end; + +instrumentation_begin (); time_start =ns_to_ktime (local_clock ());
@@ -152,13 +154,18 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,* suspended is generally unsafe. */ stop_critical_timings (); -if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) {ct_cpuidle_enter (); +/* Annotate away the indirect call */ +instrumentation_begin (); + } target_state->enter_s2idle (dev, drv, index);if (WARN_ON_ONCE (!irqs_disabled ()))raw_local_irq_disable (); -if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) { +instrumentation_end ();ct_cpuidle_exit (); + }tick_unfreeze ();start_critical_timings ();
@@ -166,6 +173,7 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv, dev->states_usage[index].s2idle_time +=ktime_us_delta (time_end, time_start); dev->states_usage[index].s2idle_usage++; +instrumentation_end (); }/**
@@ -200,8 +208,9 @@ int cpuidle_enter_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev)* @drv: cpuidle driver for this cpu * @index: index into the states table in @drv of the state to enter */ -int cpuidle_enter_state (struct cpuidle_device *dev,struct cpuidle_driver *drv, -int index) +noinstr int cpuidle_enter_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv, +int index) {int entered_state;
@@ -209,6 +218,8 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,bool broadcast = !!(target_state->flags &CPUIDLE_FLAG_TIMER_STOP );ktime_t time_start, time_end; +instrumentation_begin (); +/* * Tell the time framework to switch to a broadcast timer because our * local timer will be shut down. If a local timer is used from another
@@ -235,15 +246,21 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv, time_start =ns_to_ktime (local_clock ());stop_critical_timings (); -if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) {ct_cpuidle_enter (); +/* Annotate away the indirect call */ +instrumentation_begin (); + } entered_state = target_state->enter (dev, drv, index); +if (WARN_ONCE (!irqs_disabled (),"%ps leaked IRQ state" , target_state->enter))raw_local_irq_disable (); -if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) +if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) { +instrumentation_end ();ct_cpuidle_exit (); + }start_critical_timings ();sched_clock_idle_wakeup_event ();
@@ -306,6 +323,8 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv, dev->states_usage[index].rejected++; } +instrumentation_end (); +return entered_state; }
Peter Zijlstra
maintainerPATCH v3 48/51 cpuidle,arch: Mark all ct_cpuidle_enter() callers __cpuidle
In reply to Peter ZijlstraFor all cpuidle drivers that use CPUIDLE_FLAG_RCU_IDLE, ensure that all functions that call ct_cpuidle_enter() are marked __cpuidle.
( due to lack of noinstr validation on these platforms it is entirely
possible this isn't complete )
--- arch/arm/mach-imx/cpuidle-imx6q.c | 4 ++-- arch/arm/mach-imx/cpuidle-imx6sx.c | 4 ++-- arch/arm/mach-omap2/omap-mpuss-lowpower.c | 4 ++-- arch/arm/mach-omap2/pm34xx.c | 2 +- arch/arm64/kernel/cpuidle.c | 2 +- drivers/cpuidle/cpuidle-arm.c | 4 ++-- drivers/cpuidle/cpuidle-big_little.c | 4 ++-- drivers/cpuidle/cpuidle-mvebu-v7.c | 6 +++--- drivers/cpuidle/cpuidle-psci.c | 17 ++++++----------- drivers/cpuidle/cpuidle-qcom-spm.c | 4 ++-- drivers/cpuidle/cpuidle-riscv-sbi.c | 10 +++++----- drivers/cpuidle/cpuidle-tegra.c | 10 +++++----- 12 files changed, 33 insertions(+), 38 deletions(-)
arch/arm/mach-imx/cpuidle-imx6q.c +2 -2
@@ -17,8 +17,8 @@static int num_idle_cpus =0 ;static DEFINE_RAW_SPINLOCK (cpuidle_lock); -static int imx6q_enter_wait (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx6q_enter_wait(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {raw_spin_lock (&cpuidle_lock);if (++num_idle_cpus ==num_online_cpus ())
arch/arm/mach-imx/cpuidle-imx6sx.c +2 -2
@@ -30,8 +30,8 @@ static int imx6sx_idle_finish(unsigned lreturn 0 ; } -static int imx6sx_enter_wait (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx6sx_enter_wait(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {imx6_set_lpm (WAIT_UNCLOCKED );
arch/arm/mach-omap2/omap-mpuss-lowpower.c +2 -2
@@ -224,8 +224,8 @@ static void __init save_l2x0_context(voi* 2 - CPUx L1 and logic lost + GIC lost: MPUSS OSWR * 3 - CPUx L1 and logic lost + GIC + L2 lost: DEVICE OFF */ -int omap4_enter_lowpower (unsigned int cpu,unsigned int power_state, -bool rcuidle) +__cpuidle int omap4_enter_lowpower(unsigned int cpu, unsigned int power_state, +bool rcuidle) {struct omap4_cpu_pm_info *pm_info = &per_cpu (omap4_pm_info, cpu);unsigned int save_state =0 , cpu_logic_state =PWRDM_POWER_RET ;
arch/arm/mach-omap2/pm34xx.c +1 -1
@@ -175,7 +175,7 @@ static int omap34xx_do_sram_idle(unsignereturn 0 ; } -void omap_sram_idle (bool rcuidle) +__cpuidle void omap_sram_idle(bool rcuidle) {/* Variable to tell what needs to be saved and restored * in omap_sram_idle*/
arch/arm64/kernel/cpuidle.c +1 -1
@@ -62,7 +62,7 @@ int acpi_processor_ffh_lpi_probe(unsignereturn psci_acpi_cpu_init_idle (cpu); } -int acpi_processor_ffh_lpi_enter (struct acpi_lpi_state *lpi) +__cpuidle int acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi) {u32 state = lpi->address;
drivers/cpuidle/cpuidle-arm.c +2 -2
@@ -31,8 +31,8 @@* Called from the CPUidle framework to program the device to the * specified target state selected by the governor. */ -static int arm_enter_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int arm_enter_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) {/* * Pass idle state index to arm_cpuidle_suspend which in turn
drivers/cpuidle/cpuidle-big_little.c +2 -2
@@ -122,8 +122,8 @@ static int notrace bl_powerdown_finisher* Called from the CPUidle framework to program the device to the * specified target state selected by the governor. */ -static int bl_enter_powerdown (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int bl_enter_powerdown(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) {cpu_pm_enter ();ct_cpuidle_enter ();
drivers/cpuidle/cpuidle-mvebu-v7.c +3 -3
@@ -25,9 +25,9 @@static int (*mvebu_v7_cpu_suspend)(int ); -static int mvebu_v7_enter_idle (struct cpuidle_device *dev, -struct cpuidle_driver *drv, -int index) +static __cpuidle int mvebu_v7_enter_idle(struct cpuidle_device *dev, +struct cpuidle_driver *drv, +int index) {int ret;bool deepidle = false;
drivers/cpuidle/cpuidle-psci.c +6 -11
@@ -49,14 +49,9 @@ static inline u32 psci_get_domain_state(return __this_cpu_read (domain_state); } -static inline int psci_enter_state (int idx,u32 state) -{ -return CPU_PM_CPU_IDLE_ENTER_PARAM (psci_cpu_suspend_enter, idx, state); -} - -static int __psci_enter_domain_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx, -bool s2idle) +static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx, +bool s2idle) {struct psci_cpuidle_data *data =this_cpu_ptr (&psci_cpuidle_data);u32 *states = data->psci_states;
@@ -192,12 +187,12 @@ static void psci_idle_init_cpuhp(void)pr_warn ("Failed %d while setup cpuhp state\n" , err); } -static int psci_enter_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int psci_enter_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) {u32 *state =__this_cpu_read (psci_cpuidle_data.psci_states); -return psci_enter_state (idx, state[idx]); +return CPU_PM_CPU_IDLE_ENTER_PARAM (psci_cpu_suspend_enter, idx, state[idx]); }static const struct of_device_id psci_idle_state_match[] = {
drivers/cpuidle/cpuidle-qcom-spm.c +2 -2
@@ -58,8 +58,8 @@ static int qcom_cpu_spc(struct spm_drivereturn ret; } -static int spm_enter_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int spm_enter_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) { struct cpuidle_qcom_spm_data *data = container_of(drv, struct cpuidle_qcom_spm_data, cpuidle_driver);
drivers/cpuidle/cpuidle-riscv-sbi.c +5 -5
@@ -93,8 +93,8 @@ static int sbi_suspend(u32 state)return sbi_suspend_finisher (state,0 ,0 ); } -static int sbi_cpuidle_enter_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int sbi_cpuidle_enter_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) {u32 *states =__this_cpu_read (sbi_cpuidle_data.states);u32 state = states[idx];
@@ -106,9 +106,9 @@ static int sbi_cpuidle_enter_state(struc idx, state); } -static int __sbi_enter_domain_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx, -bool s2idle) +static __cpuidle int __sbi_enter_domain_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx, +bool s2idle) {struct sbi_cpuidle_data *data =this_cpu_ptr (&sbi_cpuidle_data);u32 *states = data->states;
drivers/cpuidle/cpuidle-tegra.c +5 -5
@@ -160,8 +160,8 @@ static int tegra_cpuidle_coupled_barrierreturn 0 ; } -static int tegra_cpuidle_state_enter (struct cpuidle_device *dev, -int index,unsigned int cpu) +static __cpuidle int tegra_cpuidle_state_enter(struct cpuidle_device *dev, +int index,unsigned int cpu) {int err;
@@ -226,9 +226,9 @@ static int tegra_cpuidle_adjust_state_inreturn index; } -static int tegra_cpuidle_enter (struct cpuidle_device *dev, -struct cpuidle_driver *drv, -int index) +static __cpuidle int tegra_cpuidle_enter(struct cpuidle_device *dev, +struct cpuidle_driver *drv, +int index) {bool do_rcu = drv->states[index].flags &CPUIDLE_FLAG_RCU_IDLE ;unsigned int cpu =cpu_logical_map (dev->cpu);
[tip: sched/core] cpuidle, arch: Mark all ct_cpuidle_enter() callers __cpuidle
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 69e26b4f43ed396868c6e7645f9eb06c3a0d2fee Gitweb: https://git.kernel.org/tip/69e26b4f43ed396868c6e7645f9eb06c3a0d2fee Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:44:02 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:17 +01:00
cpuidle, arch: Mark all ct_cpuidle_enter() callers __cpuidle
For all cpuidle drivers that use CPUIDLE_FLAG_RCU_IDLE, ensure that all functions that call ct_cpuidle_enter() are marked __cpuidle.
( due to lack of noinstr validation on these platforms it is entirely
possible this isn't complete )
--- arch/arm/mach-imx/cpuidle-imx6q.c | 4 ++-- arch/arm/mach-imx/cpuidle-imx6sx.c | 4 ++-- arch/arm/mach-omap2/omap-mpuss-lowpower.c | 4 ++-- arch/arm/mach-omap2/pm34xx.c | 2 +- arch/arm64/kernel/cpuidle.c | 2 +- drivers/cpuidle/cpuidle-arm.c | 4 ++-- drivers/cpuidle/cpuidle-big_little.c | 4 ++-- drivers/cpuidle/cpuidle-mvebu-v7.c | 6 +++--- drivers/cpuidle/cpuidle-psci.c | 17 ++++++----------- drivers/cpuidle/cpuidle-qcom-spm.c | 4 ++-- drivers/cpuidle/cpuidle-riscv-sbi.c | 10 +++++----- drivers/cpuidle/cpuidle-tegra.c | 10 +++++----- 12 files changed, 33 insertions(+), 38 deletions(-)
arch/arm/mach-imx/cpuidle-imx6q.c +2 -2
@@ -17,8 +17,8 @@static int num_idle_cpus =0 ;static DEFINE_RAW_SPINLOCK (cpuidle_lock); -static int imx6q_enter_wait (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx6q_enter_wait(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {raw_spin_lock (&cpuidle_lock);if (++num_idle_cpus ==num_online_cpus ())
arch/arm/mach-imx/cpuidle-imx6sx.c +2 -2
@@ -30,8 +30,8 @@ static int imx6sx_idle_finish(unsigned long val)return 0 ; } -static int imx6sx_enter_wait (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx6sx_enter_wait(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {imx6_set_lpm (WAIT_UNCLOCKED );
arch/arm/mach-omap2/omap-mpuss-lowpower.c +2 -2
@@ -224,8 +224,8 @@ static void __init save_l2x0_context(void)* 2 - CPUx L1 and logic lost + GIC lost: MPUSS OSWR * 3 - CPUx L1 and logic lost + GIC + L2 lost: DEVICE OFF */ -int omap4_enter_lowpower (unsigned int cpu,unsigned int power_state, -bool rcuidle) +__cpuidle int omap4_enter_lowpower(unsigned int cpu, unsigned int power_state, +bool rcuidle) {struct omap4_cpu_pm_info *pm_info = &per_cpu (omap4_pm_info, cpu);unsigned int save_state =0 , cpu_logic_state =PWRDM_POWER_RET ;
arch/arm/mach-omap2/pm34xx.c +1 -1
@@ -175,7 +175,7 @@ static int omap34xx_do_sram_idle(unsigned long save_state)return 0 ; } -void omap_sram_idle (bool rcuidle) +__cpuidle void omap_sram_idle(bool rcuidle) {/* Variable to tell what needs to be saved and restored * in omap_sram_idle*/
arch/arm64/kernel/cpuidle.c +1 -1
@@ -62,7 +62,7 @@ int acpi_processor_ffh_lpi_probe(unsigned int cpu)return psci_acpi_cpu_init_idle (cpu); } -int acpi_processor_ffh_lpi_enter (struct acpi_lpi_state *lpi) +__cpuidle int acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi) {u32 state = lpi->address;
drivers/cpuidle/cpuidle-arm.c +2 -2
@@ -31,8 +31,8 @@* Called from the CPUidle framework to program the device to the * specified target state selected by the governor. */ -static int arm_enter_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int arm_enter_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) {/* * Pass idle state index to arm_cpuidle_suspend which in turn
drivers/cpuidle/cpuidle-big_little.c +2 -2
@@ -122,8 +122,8 @@ static int notrace bl_powerdown_finisher(unsigned long arg)* Called from the CPUidle framework to program the device to the * specified target state selected by the governor. */ -static int bl_enter_powerdown (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int bl_enter_powerdown(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) {cpu_pm_enter ();ct_cpuidle_enter ();
drivers/cpuidle/cpuidle-mvebu-v7.c +3 -3
@@ -25,9 +25,9 @@static int (*mvebu_v7_cpu_suspend)(int ); -static int mvebu_v7_enter_idle (struct cpuidle_device *dev, -struct cpuidle_driver *drv, -int index) +static __cpuidle int mvebu_v7_enter_idle(struct cpuidle_device *dev, +struct cpuidle_driver *drv, +int index) {int ret;bool deepidle = false;
drivers/cpuidle/cpuidle-psci.c +6 -11
@@ -49,14 +49,9 @@ static inline u32 psci_get_domain_state(void)return __this_cpu_read (domain_state); } -static inline int psci_enter_state (int idx,u32 state) -{ -return CPU_PM_CPU_IDLE_ENTER_PARAM (psci_cpu_suspend_enter, idx, state); -} - -static int __psci_enter_domain_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx, -bool s2idle) +static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx, +bool s2idle) {struct psci_cpuidle_data *data =this_cpu_ptr (&psci_cpuidle_data);u32 *states = data->psci_states;
@@ -192,12 +187,12 @@ static void psci_idle_init_cpuhp(void)pr_warn ("Failed %d while setup cpuhp state\n" , err); } -static int psci_enter_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int psci_enter_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) {u32 *state =__this_cpu_read (psci_cpuidle_data.psci_states); -return psci_enter_state (idx, state[idx]); +return CPU_PM_CPU_IDLE_ENTER_PARAM (psci_cpu_suspend_enter, idx, state[idx]); }static const struct of_device_id psci_idle_state_match[] = {
drivers/cpuidle/cpuidle-qcom-spm.c +2 -2
@@ -58,8 +58,8 @@ static int qcom_cpu_spc(struct spm_driver_data *drv)return ret; } -static int spm_enter_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int spm_enter_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) { struct cpuidle_qcom_spm_data *data = container_of(drv, struct cpuidle_qcom_spm_data, cpuidle_driver);
drivers/cpuidle/cpuidle-riscv-sbi.c +5 -5
@@ -93,8 +93,8 @@ static int sbi_suspend(u32 state)return sbi_suspend_finisher (state,0 ,0 ); } -static int sbi_cpuidle_enter_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx) +static __cpuidle int sbi_cpuidle_enter_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx) {u32 *states =__this_cpu_read (sbi_cpuidle_data.states);u32 state = states[idx];
@@ -106,9 +106,9 @@ static int sbi_cpuidle_enter_state(struct cpuidle_device *dev, idx, state); } -static int __sbi_enter_domain_idle_state (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int idx, -bool s2idle) +static __cpuidle int __sbi_enter_domain_idle_state(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int idx, +bool s2idle) {struct sbi_cpuidle_data *data =this_cpu_ptr (&sbi_cpuidle_data);u32 *states = data->states;
drivers/cpuidle/cpuidle-tegra.c +5 -5
@@ -160,8 +160,8 @@ static int tegra_cpuidle_coupled_barrier(struct cpuidle_device *dev)return 0 ; } -static int tegra_cpuidle_state_enter (struct cpuidle_device *dev, -int index,unsigned int cpu) +static __cpuidle int tegra_cpuidle_state_enter(struct cpuidle_device *dev, +int index,unsigned int cpu) {int err;
@@ -226,9 +226,9 @@ static int tegra_cpuidle_adjust_state_index(int index, unsigned int cpu)return index; } -static int tegra_cpuidle_enter (struct cpuidle_device *dev, -struct cpuidle_driver *drv, -int index) +static __cpuidle int tegra_cpuidle_enter(struct cpuidle_device *dev, +struct cpuidle_driver *drv, +int index) {bool do_rcu = drv->states[index].flags &CPUIDLE_FLAG_RCU_IDLE ;unsigned int cpu =cpu_logical_map (dev->cpu);
Peter Zijlstra
maintainerPATCH v3 49/51 cpuidle,arch: Mark all regular cpuidle_state::enter methods __cpuidle
In reply to Peter ZijlstraFor all cpuidle drivers that do not use CPUIDLE_FLAG_RCU_IDLE (iow, the simple ones) make sure all the functions are marked __cpuidle.
( due to lack of noinstr validation on these platforms it is entirely
possible this isn't complete )
--- arch/arm/kernel/cpuidle.c | 4 ++-- arch/arm/mach-davinci/cpuidle.c | 4 ++-- arch/arm/mach-imx/cpuidle-imx5.c | 4 ++-- arch/arm/mach-imx/cpuidle-imx6sl.c | 4 ++-- arch/arm/mach-imx/cpuidle-imx7ulp.c | 4 ++-- arch/arm/mach-s3c/cpuidle-s3c64xx.c | 5 ++--- arch/mips/kernel/idle.c | 6 +++--- 7 files changed, 15 insertions(+), 16 deletions(-)
arch/arm/kernel/cpuidle.c +2 -2
@@ -26,8 +26,8 @@ static struct cpuidle_ops cpuidle_ops[NR* * Returns the index passed as parameter */ -int arm_cpuidle_simple_enter (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +__cpuidle int arm_cpuidle_simple_enter(struct cpuidle_device *dev, struct +cpuidle_driver *drv,int index) {cpu_do_idle ();
arch/arm/mach-davinci/cpuidle.c +2 -2
@@ -44,8 +44,8 @@ static void davinci_save_ddr_power(int e }/* Actual code that puts the SoC in different idle states */ -static int davinci_enter_idle (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int davinci_enter_idle(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {davinci_save_ddr_power (1 , ddr2_pdown);cpu_do_idle ();
arch/arm/mach-imx/cpuidle-imx5.c +2 -2
@@ -8,8 +8,8 @@#include <asm/system_misc.h> #include "cpuidle.h" -static int imx5_cpuidle_enter (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx5_cpuidle_enter(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {arm_pm_idle ();return index;
arch/arm/mach-imx/cpuidle-imx6sl.c +2 -2
@@ -11,8 +11,8 @@#include "common.h" #include "cpuidle.h" -static int imx6sl_enter_wait (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx6sl_enter_wait(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {imx6_set_lpm (WAIT_UNCLOCKED );/*
arch/arm/mach-imx/cpuidle-imx7ulp.c +2 -2
@@ -12,8 +12,8 @@#include "common.h" #include "cpuidle.h" -static int imx7ulp_enter_wait (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx7ulp_enter_wait(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {if (index ==1 )imx7ulp_set_lpm (ULP_PM_WAIT );
arch/arm/mach-s3c/cpuidle-s3c64xx.c +2 -3
@@ -19,9 +19,8 @@#include "regs-sys-s3c64xx.h" #include "regs-syscon-power-s3c64xx.h" -static int s3c64xx_enter_idle (struct cpuidle_device *dev, -struct cpuidle_driver *drv, -int index) +static __cpuidle int s3c64xx_enter_idle(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {unsigned long tmp;
arch/mips/kernel/idle.c +3 -3
@@ -241,7 +241,7 @@ void __init check_wait(void) } } -void arch_cpu_idle (void ) +__cpuidle void arch_cpu_idle(void) {if (cpu_wait)cpu_wait ();
@@ -249,8 +249,8 @@ void arch_cpu_idle(void)#ifdef CONFIG_CPU_IDLE -int mips_cpuidle_wait_enter (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +__cpuidle int mips_cpuidle_wait_enter(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {arch_cpu_idle ();return index;
[tip: sched/core] cpuidle,arch: Mark all regular cpuidle_state:: Enter methods __cpuidle
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 26388a7c353f7b1d3fd8a6df6452fa9773193155 Gitweb: https://git.kernel.org/tip/26388a7c353f7b1d3fd8a6df6452fa9773193155 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:44:03 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:18 +01:00
cpuidle,arch: Mark all regular cpuidle_state:: Enter methods __cpuidle
For all cpuidle drivers that do not use CPUIDLE_FLAG_RCU_IDLE (iow, the simple ones) make sure all the functions are marked __cpuidle.
( due to lack of noinstr validation on these platforms it is entirely
possible this isn't complete )
--- arch/arm/kernel/cpuidle.c | 4 ++-- arch/arm/mach-davinci/cpuidle.c | 4 ++-- arch/arm/mach-imx/cpuidle-imx5.c | 4 ++-- arch/arm/mach-imx/cpuidle-imx6sl.c | 4 ++-- arch/arm/mach-imx/cpuidle-imx7ulp.c | 4 ++-- arch/arm/mach-s3c/cpuidle-s3c64xx.c | 5 ++--- arch/mips/kernel/idle.c | 6 +++--- 7 files changed, 15 insertions(+), 16 deletions(-)
arch/arm/kernel/cpuidle.c +2 -2
@@ -26,8 +26,8 @@ static struct cpuidle_ops cpuidle_ops[NR_CPUS] __ro_after_init;* * Returns the index passed as parameter */ -int arm_cpuidle_simple_enter (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +__cpuidle int arm_cpuidle_simple_enter(struct cpuidle_device *dev, struct +cpuidle_driver *drv,int index) {cpu_do_idle ();
arch/arm/mach-davinci/cpuidle.c +2 -2
@@ -44,8 +44,8 @@ static void davinci_save_ddr_power(int enter, bool pdown) }/* Actual code that puts the SoC in different idle states */ -static int davinci_enter_idle (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int davinci_enter_idle(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {davinci_save_ddr_power (1 , ddr2_pdown);cpu_do_idle ();
arch/arm/mach-imx/cpuidle-imx5.c +2 -2
@@ -8,8 +8,8 @@#include <asm/system_misc.h> #include "cpuidle.h" -static int imx5_cpuidle_enter (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx5_cpuidle_enter(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {arm_pm_idle ();return index;
arch/arm/mach-imx/cpuidle-imx6sl.c +2 -2
@@ -11,8 +11,8 @@#include "common.h" #include "cpuidle.h" -static int imx6sl_enter_wait (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx6sl_enter_wait(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {imx6_set_lpm (WAIT_UNCLOCKED );/*
arch/arm/mach-imx/cpuidle-imx7ulp.c +2 -2
@@ -12,8 +12,8 @@#include "common.h" #include "cpuidle.h" -static int imx7ulp_enter_wait (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +static __cpuidle int imx7ulp_enter_wait(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {if (index ==1 )imx7ulp_set_lpm (ULP_PM_WAIT );
arch/arm/mach-s3c/cpuidle-s3c64xx.c +2 -3
@@ -19,9 +19,8 @@#include "regs-sys-s3c64xx.h" #include "regs-syscon-power-s3c64xx.h" -static int s3c64xx_enter_idle (struct cpuidle_device *dev, -struct cpuidle_driver *drv, -int index) +static __cpuidle int s3c64xx_enter_idle(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {unsigned long tmp;
arch/mips/kernel/idle.c +3 -3
@@ -241,7 +241,7 @@ void __init check_wait(void) } } -void arch_cpu_idle (void ) +__cpuidle void arch_cpu_idle(void) {if (cpu_wait)cpu_wait ();
@@ -249,8 +249,8 @@ void arch_cpu_idle(void)#ifdef CONFIG_CPU_IDLE -int mips_cpuidle_wait_enter (struct cpuidle_device *dev, -struct cpuidle_driver *drv,int index) +__cpuidle int mips_cpuidle_wait_enter(struct cpuidle_device *dev, +struct cpuidle_driver *drv,int index) {arch_cpu_idle ();return index;
Peter Zijlstra
maintainerPATCH v3 50/51 cpuidle: Comments about noinstr/__cpuidle
In reply to Peter ZijlstraAdd a few words on noinstr / __cpuidle usage.
--- drivers/cpuidle/cpuidle.c | 12 ++++++++++++ include/linux/compiler_types.h | 10 ++++++++++ 2 files changed, 22 insertions(+)
drivers/cpuidle/cpuidle.c +12 -0
@@ -252,6 +252,18 @@ noinstr int cpuidle_enter_state(struct cinstrumentation_begin (); } +/* +* NOTE!! +* +* For cpuidle_state::enter() methods that do *NOT* set +* CPUIDLE_FLAG_RCU_IDLE RCU will be disabled here and these functions +* must be marked either noinstr or __cpuidle. +* +* For cpuidle_state::enter() methods that *DO* set +* CPUIDLE_FLAG_RCU_IDLE this isn't required, but they must mark the +* function calling ct_cpuidle_enter() as noinstr/__cpuidle and all +* functions called within the RCU-idle region. +*/ entered_state = target_state->enter (dev, drv, index);if (WARN_ONCE (!irqs_disabled (),"%ps leaked IRQ state" , target_state->enter))
include/linux/compiler_types.h +10 -0
@@ -233,6 +233,16 @@ struct ftrace_likely_data {#define noinstr __noinstr_section(".noinstr.text") +/* +* The __cpuidle section is used twofold: +* +* 1) the original use -- identifying if a CPU is 'stuck' in idle state based +* on it's instruction pointer. See cpu_in_idle(). +* +* 2) supressing instrumentation around where cpuidle disables RCU; where the +* function isn't strictly required for #1, this is interchangeable with +* noinstr. +*/ #define __cpuidle __noinstr_section(".cpuidle.text")#endif /* __KERNEL__ */
[tip: sched/core] cpuidle: Add comments about noinstr/__cpuidle usage
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 0e985e9d22864e29d5d2b3d909ad15134d7f6d46 Gitweb: https://git.kernel.org/tip/0e985e9d22864e29d5d2b3d909ad15134d7f6d46 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:44:04 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:18 +01:00
cpuidle: Add comments about noinstr/__cpuidle usage
Add a few words on noinstr / __cpuidle usage.
--- drivers/cpuidle/cpuidle.c | 12 ++++++++++++ include/linux/compiler_types.h | 10 ++++++++++ 2 files changed, 22 insertions(+)
drivers/cpuidle/cpuidle.c +12 -0
@@ -252,6 +252,18 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev,instrumentation_begin (); } +/* +* NOTE!! +* +* For cpuidle_state::enter() methods that do *NOT* set +* CPUIDLE_FLAG_RCU_IDLE RCU will be disabled here and these functions +* must be marked either noinstr or __cpuidle. +* +* For cpuidle_state::enter() methods that *DO* set +* CPUIDLE_FLAG_RCU_IDLE this isn't required, but they must mark the +* function calling ct_cpuidle_enter() as noinstr/__cpuidle and all +* functions called within the RCU-idle region. +*/ entered_state = target_state->enter (dev, drv, index);if (WARN_ONCE (!irqs_disabled (),"%ps leaked IRQ state" , target_state->enter))
include/linux/compiler_types.h +10 -0
@@ -239,6 +239,16 @@ struct ftrace_likely_data {#define noinstr __noinstr_section(".noinstr.text") +/* +* The __cpuidle section is used twofold: +* +* 1) the original use -- identifying if a CPU is 'stuck' in idle state based +* on it's instruction pointer. See cpu_in_idle(). +* +* 2) supressing instrumentation around where cpuidle disables RCU; where the +* function isn't strictly required for #1, this is interchangeable with +* noinstr. +*/ #define __cpuidle __noinstr_section(".cpuidle.text")#endif /* __KERNEL__ */
Peter Zijlstra
maintainerPATCH v3 51/51 context_tracking: Fix noinstr vs KASAN
In reply to Peter Zijlstravmlinux.o: warning: objtool: __ct_user_enter+0x72: call to __kasan_check_write() leaves .noinstr.text section vmlinux.o: warning: objtool: __ct_user_exit+0x47: call to __kasan_check_write() leaves .noinstr.text section
--- kernel/context_tracking.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/context_tracking.c +6 -6
@@ -510,7 +510,7 @@ void noinstr __ct_user_enter(enum ctx_st* In this we case we don't care about any concurrency/ordering. */ if (!IS_ENABLED (CONFIG_CONTEXT_TRACKING_IDLE )) -atomic_set (&ct->state, state); +arch_atomic_set (&ct->state, state); }else {/* * Even if context tracking is disabled on this CPU, because it's outside
@@ -527,7 +527,7 @@ void noinstr __ct_user_enter(enum ctx_st*/ if (!IS_ENABLED (CONFIG_CONTEXT_TRACKING_IDLE )) {/* Tracking for vtime only, no concurrent RCU EQS accounting */ -atomic_set (&ct->state, state); +arch_atomic_set (&ct->state, state); }else {/* * Tracking for vtime and RCU EQS. Make sure we don't race
@@ -535,7 +535,7 @@ void noinstr __ct_user_enter(enum ctx_st* RCU only requires RCU_DYNTICKS_IDX increments to be fully * ordered. */ -atomic_add (state, &ct->state); +arch_atomic_add (state, &ct->state); } } }
@@ -630,12 +630,12 @@ void noinstr __ct_user_exit(enum ctx_sta* In this we case we don't care about any concurrency/ordering. */ if (!IS_ENABLED (CONFIG_CONTEXT_TRACKING_IDLE )) -atomic_set (&ct->state,CONTEXT_KERNEL ); +arch_atomic_set (&ct->state,CONTEXT_KERNEL ); }else {if (!IS_ENABLED (CONFIG_CONTEXT_TRACKING_IDLE )) {/* Tracking for vtime only, no concurrent RCU EQS accounting */ -atomic_set (&ct->state,CONTEXT_KERNEL ); +arch_atomic_set (&ct->state,CONTEXT_KERNEL ); }else {/* * Tracking for vtime and RCU EQS. Make sure we don't race
@@ -643,7 +643,7 @@ void noinstr __ct_user_exit(enum ctx_sta* RCU only requires RCU_DYNTICKS_IDX increments to be fully * ordered. */ -atomic_sub (state, &ct->state); +arch_atomic_sub (state, &ct->state); } } }
[tip: sched/core] context_tracking: Fix noinstr vs KASAN
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 0e26e1de0032779e43929174339429c16307a299 Gitweb: https://git.kernel.org/tip/0e26e1de0032779e43929174339429c16307a299 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 12 Jan 2023 20:44:05 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 13 Jan 2023 11:48:18 +01:00
context_tracking: Fix noinstr vs KASAN
Low level noinstr context-tracking code is calling out to instrumented code on KASAN:
vmlinux.o: warning: objtool: __ct_user_enter+0x72: call to __kasan_check_write() leaves .noinstr.text section vmlinux.o: warning: objtool: __ct_user_exit+0x47: call to __kasan_check_write() leaves .noinstr.text section
Use even lower level atomic methods to avoid the instrumentation.
--- kernel/context_tracking.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/context_tracking.c +6 -6
@@ -510,7 +510,7 @@ void noinstr __ct_user_enter(enum ctx_state state)* In this we case we don't care about any concurrency/ordering. */ if (!IS_ENABLED (CONFIG_CONTEXT_TRACKING_IDLE )) -atomic_set (&ct->state, state); +arch_atomic_set (&ct->state, state); }else {/* * Even if context tracking is disabled on this CPU, because it's outside
@@ -527,7 +527,7 @@ void noinstr __ct_user_enter(enum ctx_state state)*/ if (!IS_ENABLED (CONFIG_CONTEXT_TRACKING_IDLE )) {/* Tracking for vtime only, no concurrent RCU EQS accounting */ -atomic_set (&ct->state, state); +arch_atomic_set (&ct->state, state); }else {/* * Tracking for vtime and RCU EQS. Make sure we don't race
@@ -535,7 +535,7 @@ void noinstr __ct_user_enter(enum ctx_state state)* RCU only requires RCU_DYNTICKS_IDX increments to be fully * ordered. */ -atomic_add (state, &ct->state); +arch_atomic_add (state, &ct->state); } } }
@@ -630,12 +630,12 @@ void noinstr __ct_user_exit(enum ctx_state state)* In this we case we don't care about any concurrency/ordering. */ if (!IS_ENABLED (CONFIG_CONTEXT_TRACKING_IDLE )) -atomic_set (&ct->state,CONTEXT_KERNEL ); +arch_atomic_set (&ct->state,CONTEXT_KERNEL ); }else {if (!IS_ENABLED (CONFIG_CONTEXT_TRACKING_IDLE )) {/* Tracking for vtime only, no concurrent RCU EQS accounting */ -atomic_set (&ct->state,CONTEXT_KERNEL ); +arch_atomic_set (&ct->state,CONTEXT_KERNEL ); }else {/* * Tracking for vtime and RCU EQS. Make sure we don't race
@@ -643,7 +643,7 @@ void noinstr __ct_user_exit(enum ctx_state state)* RCU only requires RCU_DYNTICKS_IDX increments to be fully * ordered. */ -atomic_sub (state, &ct->state); +arch_atomic_sub (state, &ct->state); } } }
Quoted 14 lines · Peter Zijlstra
Hi All!
The (hopefully) final respin of cpuidle vs rcu cleanup patches. Barring any objections I'll be queueing these patches in tip/sched/core in the next few days.
v2: https://lkml.kernel.org/r/[email protected]
These here patches clean up the mess that is cpuidle vs rcuidle.
At the end of the ride there's only on
RCU_NONIDLEuser left:arch/arm64/kernel/suspend.c: RCU_NONIDLE(__cpu_suspend_exit());
Quoted 7 lines
And I know Mark has been prodding that with something sharp.
The last version was tested by a number of people and I'm hoping to not have broken anything in the meantime ;-)
Changes since v2:
150 rcutorture hours on each of the default scenarios passed. This is qemu/KVM on x86:
Quoted 157 lines
- rebased to v6.2-rc3; as available at:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.gitsched/idle- folded: https://lkml.kernel.org/r/[email protected]
which makes the ARM cpuidle index 0 consistently not useCPUIDLE_FLAG_RCU_IDLE, as requested by Ulf.- added a few more
__always_inlineto empty stub functions as found by the robot.- Used _RET_IP_ instead of _THIS_IP_ in a few placed because of:
https://github.com/ClangBuiltLinux/linux/issues/263- Added new patches to address various robot reports:
#35: trace,hardirq: No moar _rcuidle() tracing #47: cpuidle: Ensure ct_cpuidle_enter() is always called from noinstr/__cpuidle #48: cpuidle,arch: Mark all ct_cpuidle_enter() callers __cpuidle #49: cpuidle,arch: Mark all regular cpuidle_state::enter methods __cpuidle #50: cpuidle: Comments about noinstr/__cpuidle #51: context_tracking: Fix noinstr vs KASAN--- arch/alpha/kernel/process.c | 1 - arch/alpha/kernel/vmlinux.lds.S | 1 - arch/arc/kernel/process.c | 3 ++ arch/arc/kernel/vmlinux.lds.S | 1 - arch/arm/include/asm/vmlinux.lds.h | 1 - arch/arm/kernel/cpuidle.c | 4 +- arch/arm/kernel/process.c | 1 - arch/arm/kernel/smp.c | 6 +-- arch/arm/mach-davinci/cpuidle.c | 4 +- arch/arm/mach-gemini/board-dt.c | 3 +- arch/arm/mach-imx/cpuidle-imx5.c | 4 +- arch/arm/mach-imx/cpuidle-imx6q.c | 8 ++-- arch/arm/mach-imx/cpuidle-imx6sl.c | 4 +- arch/arm/mach-imx/cpuidle-imx6sx.c | 9 ++-- arch/arm/mach-imx/cpuidle-imx7ulp.c | 4 +- arch/arm/mach-omap2/common.h | 6 ++- arch/arm/mach-omap2/cpuidle34xx.c | 16 ++++++- arch/arm/mach-omap2/cpuidle44xx.c | 29 +++++++------ arch/arm/mach-omap2/omap-mpuss-lowpower.c | 12 +++++- arch/arm/mach-omap2/pm.h | 2 +- arch/arm/mach-omap2/pm24xx.c | 51 +--------------------- arch/arm/mach-omap2/pm34xx.c | 14 +++++-- arch/arm/mach-omap2/pm44xx.c | 2 +- arch/arm/mach-omap2/powerdomain.c | 10 ++--- arch/arm/mach-s3c/cpuidle-s3c64xx.c | 5 +-- arch/arm64/kernel/cpuidle.c | 2 +- arch/arm64/kernel/idle.c | 1 - arch/arm64/kernel/smp.c | 4 +- arch/arm64/kernel/vmlinux.lds.S | 1 - arch/csky/kernel/process.c | 1 - arch/csky/kernel/smp.c | 2 +- arch/csky/kernel/vmlinux.lds.S | 1 - arch/hexagon/kernel/process.c | 1 - arch/hexagon/kernel/vmlinux.lds.S | 1 - arch/ia64/kernel/process.c | 1 + arch/ia64/kernel/vmlinux.lds.S | 1 - arch/loongarch/kernel/idle.c | 1 + arch/loongarch/kernel/vmlinux.lds.S | 1 - arch/m68k/kernel/vmlinux-nommu.lds | 1 - arch/m68k/kernel/vmlinux-std.lds | 1 - arch/m68k/kernel/vmlinux-sun3.lds | 1 - arch/microblaze/kernel/process.c | 1 - arch/microblaze/kernel/vmlinux.lds.S | 1 - arch/mips/kernel/idle.c | 14 +++---- arch/mips/kernel/vmlinux.lds.S | 1 - arch/nios2/kernel/process.c | 1 - arch/nios2/kernel/vmlinux.lds.S | 1 - arch/openrisc/kernel/process.c | 1 + arch/openrisc/kernel/vmlinux.lds.S | 1 - arch/parisc/kernel/process.c | 2 - arch/parisc/kernel/vmlinux.lds.S | 1 - arch/powerpc/kernel/idle.c | 5 +-- arch/powerpc/kernel/vmlinux.lds.S | 1 - arch/riscv/kernel/process.c | 1 - arch/riscv/kernel/vmlinux-xip.lds.S | 1 - arch/riscv/kernel/vmlinux.lds.S | 1 - arch/s390/kernel/idle.c | 1 - arch/s390/kernel/vmlinux.lds.S | 1 - arch/sh/kernel/idle.c | 1 + arch/sh/kernel/vmlinux.lds.S | 1 - arch/sparc/kernel/leon_pmc.c | 4 ++ arch/sparc/kernel/process_32.c | 1 - arch/sparc/kernel/process_64.c | 3 +- arch/sparc/kernel/vmlinux.lds.S | 1 - arch/um/kernel/dyn.lds.S | 1 - arch/um/kernel/process.c | 1 - arch/um/kernel/uml.lds.S | 1 - arch/x86/boot/compressed/vmlinux.lds.S | 1 + arch/x86/coco/tdx/tdcall.S | 15 +------ arch/x86/coco/tdx/tdx.c | 25 ++++------- arch/x86/events/amd/brs.c | 13 +++--- arch/x86/include/asm/fpu/xcr.h | 4 +- arch/x86/include/asm/irqflags.h | 11 ++--- arch/x86/include/asm/mwait.h | 14 +++---- arch/x86/include/asm/nospec-branch.h | 2 +- arch/x86/include/asm/paravirt.h | 6 ++- arch/x86/include/asm/perf_event.h | 2 +- arch/x86/include/asm/shared/io.h | 4 +- arch/x86/include/asm/shared/tdx.h | 1 - arch/x86/include/asm/special_insns.h | 8 ++-- arch/x86/include/asm/xen/hypercall.h | 2 +- arch/x86/kernel/cpu/bugs.c | 2 +- arch/x86/kernel/fpu/core.c | 4 +- arch/x86/kernel/paravirt.c | 14 ++++++- arch/x86/kernel/process.c | 65 ++++++++++++++-------------- arch/x86/kernel/vmlinux.lds.S | 1 - arch/x86/lib/memcpy_64.S | 5 +-- arch/x86/lib/memmove_64.S | 4 +- arch/x86/lib/memset_64.S | 4 +- arch/x86/xen/enlighten_pv.c | 2 +- arch/x86/xen/irq.c | 2 +- arch/xtensa/kernel/process.c | 1 + arch/xtensa/kernel/vmlinux.lds.S | 1 - drivers/acpi/processor_idle.c | 28 ++++++++----- drivers/base/power/runtime.c | 24 +++++------ drivers/clk/clk.c | 8 ++-- drivers/cpuidle/cpuidle-arm.c | 4 +- drivers/cpuidle/cpuidle-big_little.c | 12 ++++-- drivers/cpuidle/cpuidle-mvebu-v7.c | 13 ++++-- drivers/cpuidle/cpuidle-psci.c | 26 +++++------- drivers/cpuidle/cpuidle-qcom-spm.c | 4 +- drivers/cpuidle/cpuidle-riscv-sbi.c | 19 +++++---- drivers/cpuidle/cpuidle-tegra.c | 31 +++++++++----- drivers/cpuidle/cpuidle.c | 70 ++++++++++++++++++++++--------- drivers/cpuidle/dt_idle_states.c | 2 +- drivers/cpuidle/poll_state.c | 10 ++++- drivers/idle/intel_idle.c | 19 ++++----- drivers/perf/arm_pmu.c | 11 +---- drivers/perf/riscv_pmu_sbi.c | 8 +--- include/asm-generic/vmlinux.lds.h | 9 ++-- include/linux/clockchips.h | 4 +- include/linux/compiler_types.h | 18 +++++++- include/linux/cpu.h | 3 -- include/linux/cpuidle.h | 32 ++++++++++++++ include/linux/cpumask.h | 4 +- include/linux/percpu-defs.h | 2 +- include/linux/sched/idle.h | 40 +++++++++++++----- include/linux/thread_info.h | 18 +++++++- include/linux/tracepoint.h | 15 ++++++- kernel/context_tracking.c | 12 +++--- kernel/cpu_pm.c | 9 ---- kernel/printk/printk.c | 2 +- kernel/sched/idle.c | 47 ++++++--------------- kernel/time/tick-broadcast-hrtimer.c | 29 ++++++------- kernel/time/tick-broadcast.c | 6 ++- kernel/trace/trace.c | 3 ++ kernel/trace/trace_preemptirq.c | 50 ++++++---------------- lib/ubsan.c | 5 ++- mm/kasan/kasan.h | 4 ++ mm/kasan/shadow.c | 38 +++++++++++++++++ tools/objtool/check.c | 17 ++++++++ 131 files changed, 617 insertions(+), 523 deletions(-)