rcu/context-tracking: Merge RCU eqs-dynticks counter to context tracking v2
Subsystems (23)
Frederic Weisbecker
maintainercover rcu/context-tracking: Merge RCU eqs-dynticks counter to context tracking v2
This mixes up the RCU dynticks counter and the context tracking state updates into a single atomic instruction. This may serve several purposes:
1) Improve CPU isolation with deferring some disturbances until sensitive userspace workload completes and goes to the kernel. This can take several forms, for example smp_call_function_housekeeping() or on_each_housekeeping_cpu() to enqueue and execute work on all housekeeping CPUs. Then an atomic operation on ct->state can defer the work on nohz_full CPUs until they run in kernel (or IPI them if they are in kernel mode), see this proposal by Peter:
https://lore.kernel.org/all/[email protected]/#r
2) Unearth sysidle (https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/commit/?h=sysidle.2017.05.11a&id=fe5ac724d81a3c7803e60c2232718f212f3f38d4)
This feature allowed to shutdown the tick on the last housekeeping
CPU once the rest of the system is fully idle. We needed some proper
fully ordered context tracking for that.
Inspired by Peterz: https://lore.kernel.org/all/[email protected]
Passed 100 hours of TREE01.
---
Changes since v1 after Paul, Nicolas and Peter's reviews:
* Rebased against latest rcu:dev
* Remove context_tracking_in_user()
* Tag rcu_idle_enter() as noinstr
* Add comments to explain why some RCU and context tracking functions can not be tagged as noinstr yet even though they should
* Add Acks from Paul
* Rename context_tracking_cpu_set() to ct_cpu_track_user()
* Some changelogs fixed
* Fix missing renames in cpuidle
* Add missing header comments for ct_irq_*()
* Add accessors for ->dynticks_nesting and ->dynticks_nmi_nesting
* Fix ->dynticks_nesting[_nmi] that were printed from the local ct instead of the remote one
* Only build rcu_irq_exit_check_preempt() when CONFIG_NO_HZ_FULL=y
* Remove static from rcu_preempt_deferred_qs() for all configs
* CONTEXT_* values now count from 0 to 4 instead of using bit flags, since all these states are mutually exclusive anyway. Therefore RCU_DYNTICKS_IDX now starts at 4 instead of 8.
* Keep the lowest bits of rcu_dynticks_snap() and expand a bit the printing size accordingly.
---git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
rcu/context-tracking-v2
HEAD: 0364ddb8830db4d4b78f7bfd9344e12d22e9cd32
Thanks,
Frederic
---
Frederic Weisbecker (21):
context_tracking: Remove unused context_tracking_in_user()
rcu: Set rcu_idle_enter() as noinstr
rcu: Add a note about noinstr VS unsafe eqs functions
context_tracking: Add a note about noinstr VS unsafe context tracking functions
context_tracking: Rename __context_tracking_enter/exit() to __ct_user_enter/exit()
context_tracking: Rename context_tracking_user_enter/exit() to user_enter/exit_callable()
context_tracking: Rename context_tracking_enter/exit() to ct_user_enter/exit()
context_tracking: Rename context_tracking_cpu_set() to ct_cpu_track_user()
context_tracking: Split user tracking Kconfig
context_tracking: Take idle eqs entrypoints over RCU
context_tracking: Take IRQ eqs entrypoints over RCU
context_tracking: Take NMI eqs entrypoints over RCU
rcu/context-tracking: Remove rcu_irq_enter/exit()
rcu/context_tracking: Move dynticks counter to context tracking rcu/context_tracking: Move dynticks_nesting to context tracking rcu/context_tracking: Move dynticks_nmi_nesting to context tracking rcu/context-tracking: Move deferred nocb resched to context tracking rcu/context-tracking: Move RCU-dynticks internal functions to context_tracking rcu/context-tracking: Remove unused and/or unecessary middle functions
context_tracking: Convert state to atomic_t
rcu/context_tracking: Merge dynticks counter and context tracking states
.../RCU/Design/Requirements/Requirements.rst | 10 +- Documentation/RCU/stallwarn.rst | 6 +- .../time/context-tracking/arch-support.txt | 6 +- arch/Kconfig | 8 +- arch/arm/Kconfig | 2 +- arch/arm/kernel/entry-common.S | 4 +- arch/arm/kernel/entry-header.S | 12 +- arch/arm/mach-imx/cpuidle-imx6q.c | 5 +- arch/arm64/Kconfig | 2 +- arch/arm64/kernel/entry-common.c | 14 +- arch/csky/Kconfig | 2 +- arch/csky/kernel/entry.S | 8 +- arch/mips/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/powerpc/include/asm/context_tracking.h | 2 +- arch/riscv/Kconfig | 2 +- arch/riscv/kernel/entry.S | 12 +- arch/sparc/Kconfig | 2 +- arch/sparc/kernel/rtrap_64.S | 2 +- arch/x86/Kconfig | 4 +- arch/x86/mm/fault.c | 2 +- drivers/acpi/processor_idle.c | 5 +- drivers/cpuidle/cpuidle-psci.c | 8 +- drivers/cpuidle/cpuidle-riscv-sbi.c | 8 +- drivers/cpuidle/cpuidle.c | 9 +- include/linux/context_tracking.h | 89 ++-- include/linux/context_tracking_irq.h | 21 + include/linux/context_tracking_state.h | 103 +++- include/linux/entry-common.h | 10 +- include/linux/hardirq.h | 12 +- include/linux/rcupdate.h | 15 +- include/linux/rcutiny.h | 6 - include/linux/rcutree.h | 15 +- include/linux/tracepoint.h | 4 +- init/Kconfig | 4 +- kernel/context_tracking.c | 556 +++++++++++++++++++-- kernel/cpu_pm.c | 8 +- kernel/entry/common.c | 16 +- kernel/extable.c | 4 +- kernel/locking/lockdep.c | 2 +- kernel/rcu/Kconfig | 2 + kernel/rcu/rcu.h | 4 - kernel/rcu/tree.c | 476 +----------------- kernel/rcu/tree.h | 8 - kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_plugin.h | 38 +- kernel/rcu/tree_stall.h | 8 +- kernel/rcu/update.c | 2 +- kernel/sched/core.c | 2 +- kernel/sched/idle.c | 10 +- kernel/sched/sched.h | 1 + kernel/softirq.c | 4 +- kernel/time/Kconfig | 22 +- kernel/time/tick-sched.c | 2 +- kernel/trace/trace.c | 8 +- 55 files changed, 845 insertions(+), 748 deletions(-)
Frederic Weisbecker
maintainerPATCH 1/21 context_tracking: Remove unused context_tracking_in_user()
This function is not used and CT_WARN_ON() coupled with ct_state() is the preferred way to assert context tracking state values.
--- include/linux/context_tracking_state.h | 5 ----- 1 file changed, 5 deletions(-)
include/linux/context_tracking_state.h +0 -5
@@ -41,12 +41,7 @@ static inline bool context_tracking_enabled_this_cpu(void) return context_tracking_enabled() && __this_cpu_read(context_tracking.active); } -static __always_inline bool context_tracking_in_user(void) -{ -return __this_cpu_read (context_tracking.state) ==CONTEXT_USER ; -}#else -static inline bool context_tracking_in_user (void ) {return false; }static inline bool context_tracking_enabled (void ) {return false; }static inline bool context_tracking_enabled_cpu (int cpu) {return false; }static inline bool context_tracking_enabled_this_cpu (void ) {return false; }
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 2/21 rcu: Set rcu_idle_enter() as noinstr
In reply to Frederic WeisbeckerThis function enters into RCU idle mode, make sure that illegal uses of RCU are spotted there by objtools.
--- kernel/rcu/tree.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -659,7 +659,7 @@ static noinstr void rcu_eqs_enter(bool user)* If you add or remove a call to rcu_idle_enter(), be sure to test with * CONFIG_RCU_EQS_DEBUG=y. */ -void rcu_idle_enter (void ) +noinstr void rcu_idle_enter(void) {lockdep_assert_irqs_disabled ();rcu_eqs_enter (false);
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 3/21 rcu: Add a note about noinstr VS unsafe eqs functions
In reply to Frederic WeisbeckerSome RCU functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations.
Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing.
In any case and until this get solved, those RCU functions can't get tagged as noinstr even though they should.
--- kernel/rcu/tree.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/rcu/tree.c +12 -0
@@ -838,6 +838,10 @@ void rcu_irq_exit_check_preempt(void)* * If you add or remove a call to rcu_irq_exit_irqson(), be sure to test * with CONFIG_RCU_EQS_DEBUG=y. +* +* FIXME: This function should be noinstr but the below local_irq_restore() is +* unsafe because it involves illegal RCU uses through tracing and lockdep. +* This must be fixed first. */ void rcu_irq_exit_irqson (void ) {
@@ -895,6 +899,10 @@ static void noinstr rcu_eqs_exit(bool user)* * If you add or remove a call to rcu_idle_exit(), be sure to test with * CONFIG_RCU_EQS_DEBUG=y. +* +* FIXME: This function should be noinstr but the below local_irq_save() is +* unsafe because it involves illegal RCU uses through tracing and lockdep. +* This must be fixed first. */ void rcu_idle_exit (void ) {
@@ -1076,6 +1084,10 @@ noinstr void rcu_irq_enter(void)* * If you add or remove a call to rcu_irq_enter_irqson(), be sure to test * with CONFIG_RCU_EQS_DEBUG=y. +* +* FIXME: This function should be noinstr but the below local_irq_save() is +* unsafe because it involves illegal RCU uses through tracing and lockdep. +* This must be fixed first. */ void rcu_irq_enter_irqson (void ) {
Signature · 1 line
2.25.1
Peter Zijlstra
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
Quoted 10 lines · Frederic Weisbecker
Some RCU functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations.
Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing.
In any case and until this get solved, those RCU functions can't get tagged as noinstr even though they should.
No need to speculate on how to solve it; the generic entry code has alternatives that are good. It's just that reworking your architecture isn't trivial.
Frederic Weisbecker
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Frederic Weisbecker
Some RCU functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations.
Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing.
Quoted 6 lines
Quoted 2 lines
In any case and until this get solved, those RCU functions can't get tagged as noinstr even though they should.
No need to speculate on how to solve it; the generic entry code has alternatives that are good. It's just that reworking your architecture isn't trivial.
Speaking of, please have a look at this:
https://lore.kernel.org/all/20220518162118.GA2661055@paulmck-ThinkPad-P17-Gen-1/
Peter Zijlstra
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
In reply to Frederic WeisbeckerQuoted 10 lines · Frederic Weisbecker
Some RCU functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations.
Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing.
In any case and until this get solved, those RCU functions can't get tagged as noinstr even though they should.
FWIW, this also means RCU_NONIDLE() is broken. Luckily there aren't many users left.
Paul E. McKenney
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
Some RCU functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations.
Quoted 8 lines
Quoted 5 lines
Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing.
In any case and until this get solved, those RCU functions can't get tagged as noinstr even though they should.
FWIW, this also means
RCU_NONIDLE()is broken. Luckily there aren't many users left.
More to the point, rcu_irq_enter_irqson() and rcu_irq_exit_irqson() need the same help that rcu_idle_enter() and rcu_idle_exit() got earlier this week [1], independent of RCU_NONIDLE().
Unless Frederic beats me to it, I will queue something up.
Thanx, Paul
[1] https://lore.kernel.org/all/20220518162118.GA2661055@paulmck-ThinkPad-P17-Gen-1/
Frederic Weisbecker
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
Quoted 13 lines · Paul E. McKenney
Quoted 12 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Some RCU functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations.
Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing.
In any case and until this get solved, those RCU functions can't get tagged as noinstr even though they should.
Quoted 8 lines
Quoted 2 lines
FWIW, this also means
RCU_NONIDLE()is broken. Luckily there aren't many users left.More to the point,
rcu_irq_enter_irqson()andrcu_irq_exit_irqson()need the same help thatrcu_idle_enter()andrcu_idle_exit()got earlier this week [1], independent ofRCU_NONIDLE().Unless Frederic beats me to it, I will queue something up.
It's in the patchset I just posted (02/21).
Quoted 4 lines
Thanx, Paul
[1] https://lore.kernel.org/all/20220518162118.GA2661055@paulmck-ThinkPad-P17-Gen-1/
Paul E. McKenney
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
Quoted 17 lines · Frederic Weisbecker
Quoted 16 lines · Paul E. McKenney
Quoted 14 lines · Peter Zijlstra
Quoted 10 lines · Frederic Weisbecker
Some RCU functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations.
Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing.
In any case and until this get solved, those RCU functions can't get tagged as noinstr even though they should.
FWIW, this also means
RCU_NONIDLE()is broken. Luckily there aren't many users left.
Quoted 7 lines
Quoted 5 lines
More to the point,
rcu_irq_enter_irqson()andrcu_irq_exit_irqson()need the same help thatrcu_idle_enter()andrcu_idle_exit()got earlier this week [1], independent ofRCU_NONIDLE().Unless Frederic beats me to it, I will queue something up.
It's in the patchset I just posted (02/21).
Even better!!! Thank you!
Thanx, Paul
Peter Zijlstra
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
In reply to Frederic Weisbeckerkernel/rcu/tree.c static void noinstr rcu_eqs_exit(bool user) · line 908
Quoted 11 lines · Frederic Weisbecker
kernel/rcu/tree.c +4
-0@@ -895,6 +899,10 @@ static void noinstr rcu_eqs_exit(bool user)* * If you add or remove a call to rcu_idle_exit(), be sure to test with * CONFIG_RCU_EQS_DEBUG=y. +* +* FIXME: This function should be noinstr but the below local_irq_save() is +* unsafe because it involves illegal RCU uses through tracing and lockdep. +* This must be fixed first. */ void rcu_idle_exit (void ) {
Urgh, except this one... I'm sure I fixed that at some point. Clearly that never made it in :/
Frederic Weisbecker
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
Quoted 15 lines · Peter Zijlstra
kernel/rcu/tree.c static void noinstr rcu_eqs_exit(bool user) · line 908
Quoted 11 lines · Frederic Weisbecker
kernel/rcu/tree.c +4
-0@@ -895,6 +899,10 @@ static void noinstr rcu_eqs_exit(bool user)* * If you add or remove a call to rcu_idle_exit(), be sure to test with * CONFIG_RCU_EQS_DEBUG=y. +* +* FIXME: This function should be noinstr but the below local_irq_save() is +* unsafe because it involves illegal RCU uses through tracing and lockdep. +* This must be fixed first. */ void rcu_idle_exit (void ) {Urgh, except this one... I'm sure I fixed that at some point. Clearly that never made it in :/
Most definetly the trickiest one...
Peter Zijlstra
maintainerRe: [PATCH 03/21] rcu: Add a note about noinstr VS unsafe eqs functions
In reply to Peter ZijlstraQuoted 15 lines · Peter Zijlstra
kernel/rcu/tree.c static void noinstr rcu_eqs_exit(bool user) · line 908
Quoted 11 lines · Frederic Weisbecker
kernel/rcu/tree.c +4
-0@@ -895,6 +899,10 @@ static void noinstr rcu_eqs_exit(bool user)* * If you add or remove a call to rcu_idle_exit(), be sure to test with * CONFIG_RCU_EQS_DEBUG=y. +* +* FIXME: This function should be noinstr but the below local_irq_save() is +* unsafe because it involves illegal RCU uses through tracing and lockdep. +* This must be fixed first. */ void rcu_idle_exit (void ) {Urgh, except this one... I'm sure I fixed that at some point. Clearly that never made it in :/
I found my patches and brushed them off, see here:
Frederic Weisbecker
maintainerPATCH 4/21 context_tracking: Add a note about noinstr VS unsafe context tracking functions
In reply to Frederic WeisbeckerSome context tracking functions enter or exit into/from RCU idle mode while using trace-able and lockdep-aware IRQs (un-)masking. This could be easily solved with using raw versions of local_irq_*() but we would then lose some precious debugging informations.
Another possible way to solve this may consist in using rude RCU-tasks in lockdep and irqsoff tracing.
In any case and until this get solved, those functions can't get tagged as noinstr even though they should.
--- kernel/context_tracking.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
kernel/context_tracking.c +18 -0
@@ -103,6 +103,11 @@ void noinstr __context_tracking_enter(enum ctx_state state) }EXPORT_SYMBOL_GPL (__context_tracking_enter); +/* +* FIXME: This function should be noinstr but the below local_irq_restore() is +* unsafe because it involves illegal RCU uses through tracing and lockdep. This +* must be fixed first. +*/ void context_tracking_enter (enum ctx_state state) {unsigned long flags;
@@ -125,6 +130,10 @@ void context_tracking_enter(enum ctx_state state)NOKPROBE_SYMBOL (context_tracking_enter);EXPORT_SYMBOL_GPL (context_tracking_enter); +/* +* FIXME: This function should be noinstr but it unsafely calls local_irq_restore(), +* involving illegal RCU uses through tracing and lockdep. This must be fixed first. +*/ void context_tracking_user_enter (void ) {user_enter ();
@@ -168,6 +177,11 @@ void noinstr __context_tracking_exit(enum ctx_state state) }EXPORT_SYMBOL_GPL (__context_tracking_exit); +/* +* FIXME: This function should be noinstr but the below local_irq_save() is +* unsafe because it involves illegal RCU uses through tracing and lockdep. This +* must be fixed first. +*/ void context_tracking_exit (enum ctx_state state) {unsigned long flags;
@@ -182,6 +196,10 @@ void context_tracking_exit(enum ctx_state state)NOKPROBE_SYMBOL (context_tracking_exit);EXPORT_SYMBOL_GPL (context_tracking_exit); +/* +* FIXME: This function should be noinstr but it unsafely calls local_irq_save(), +* involving illegal RCU uses through tracing and lockdep. This must be fixed first. +*/ void context_tracking_user_exit (void ) {user_exit ();
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 5/21 context_tracking: Rename __context_tracking_enter/exit() to __ct_user_enter/exit()
In reply to Frederic WeisbeckerThe context tracking namespace is going to expand and some new functions will require even longer names. Start shrinking the context_tracking prefix to "ct" as is already the case for some existing macros, this will make the introduction of new functions easier.
--- include/linux/context_tracking.h | 12 ++++++------ kernel/context_tracking.c | 20 ++++++++++---------- 2 files changed, 16 insertions(+), 16 deletions(-)
include/linux/context_tracking.h +6 -6
@@ -14,8 +14,8 @@extern void context_tracking_cpu_set (int cpu);/* Called with interrupts disabled. */ -extern void __context_tracking_enter (enum ctx_state state); -extern void __context_tracking_exit (enum ctx_state state); +extern void __ct_user_enter (enum ctx_state state); +extern void __ct_user_exit (enum ctx_state state);extern void context_tracking_enter (enum ctx_state state);extern void context_tracking_exit (enum ctx_state state);
@@ -38,13 +38,13 @@ static inline void user_exit(void) static __always_inline void user_enter_irqoff(void) {if (context_tracking_enabled ()) -__context_tracking_enter (CONTEXT_USER ); +__ct_user_enter (CONTEXT_USER ); } static __always_inline void user_exit_irqoff(void) {if (context_tracking_enabled ()) -__context_tracking_exit (CONTEXT_USER ); +__ct_user_exit (CONTEXT_USER ); }static inline enum ctx_state exception_enter (void )
@@ -74,7 +74,7 @@ static inline void exception_exit(enum ctx_state prev_ctx) static __always_inline bool context_tracking_guest_enter(void) {if (context_tracking_enabled ()) -__context_tracking_enter (CONTEXT_GUEST ); +__ct_user_enter (CONTEXT_GUEST );return context_tracking_enabled_this_cpu (); }
@@ -82,7 +82,7 @@ static __always_inline bool context_tracking_guest_enter(void) static __always_inline void context_tracking_guest_exit(void) {if (context_tracking_enabled ()) -__context_tracking_exit (CONTEXT_GUEST ); +__ct_user_exit (CONTEXT_GUEST ); }/**
kernel/context_tracking.c +10 -10
@@ -51,15 +51,15 @@ static __always_inline void context_tracking_recursion_exit(void) }/** -* context_tracking_enter - Inform the context tracking that the CPU is going -* enter user or guest space mode. +* __ct_user_enter - Inform the context tracking that the CPU is going +* to enter user or guest space mode. * * This function must be called right before we switch from the kernel * to user or guest space, when it's guaranteed the remaining kernel * instructions to execute won't use any RCU read side critical section * because this function sets RCU in extended quiescent state. */ -void noinstr __context_tracking_enter(enum ctx_state state) +void noinstr __ct_user_enter(enum ctx_state state) {/* Kernel threads aren't supposed to go to userspace */ WARN_ON_ONCE (!current->mm);
@@ -101,7 +101,7 @@ void noinstr __context_tracking_enter(enum ctx_state state) }context_tracking_recursion_exit (); } -EXPORT_SYMBOL_GPL (__context_tracking_enter); +EXPORT_SYMBOL_GPL (__ct_user_enter);/* * FIXME: This function should be noinstr but the below local_irq_restore() is
@@ -124,7 +124,7 @@ void context_tracking_enter(enum ctx_state state)return ;local_irq_save (flags); -__context_tracking_enter (state); +__ct_user_enter (state);local_irq_restore (flags); }NOKPROBE_SYMBOL (context_tracking_enter);
@@ -141,8 +141,8 @@ void context_tracking_user_enter(void)NOKPROBE_SYMBOL (context_tracking_user_enter);/** -* context_tracking_exit - Inform the context tracking that the CPU is -* exiting user or guest mode and entering the kernel. +* __ct_user_exit - Inform the context tracking that the CPU is +* exiting user or guest mode and entering the kernel. * * This function must be called after we entered the kernel from user or * guest space before any use of RCU read side critical section. This
@@ -152,7 +152,7 @@ NOKPROBE_SYMBOL(context_tracking_user_enter);* This call supports re-entrancy. This way it can be called from any exception * handler without needing to know if we came from userspace or not. */ -void noinstr __context_tracking_exit(enum ctx_state state) +void noinstr __ct_user_exit(enum ctx_state state) {if (!context_tracking_recursion_enter ())return ;
@@ -175,7 +175,7 @@ void noinstr __context_tracking_exit(enum ctx_state state) }context_tracking_recursion_exit (); } -EXPORT_SYMBOL_GPL (__context_tracking_exit); +EXPORT_SYMBOL_GPL (__ct_user_exit);/* * FIXME: This function should be noinstr but the below local_irq_save() is
@@ -190,7 +190,7 @@ void context_tracking_exit(enum ctx_state state)return ;local_irq_save (flags); -__context_tracking_exit (state); +__ct_user_exit (state);local_irq_restore (flags); }NOKPROBE_SYMBOL (context_tracking_exit);
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 6/21 context_tracking: Rename context_tracking_user_enter/exit() to user_enter/exit_callable()
In reply to Frederic Weisbeckercontext_tracking_user_enter() and context_tracking_user_exit() are ASM callable versions of user_enter() and user_exit() for architectures that didn't manage to check the context tracking static key from ASM. Change those function names to better reflect their purpose.
--- arch/arm/kernel/entry-header.S | 8 ++++---- arch/csky/kernel/entry.S | 4 ++-- arch/riscv/kernel/entry.S | 6 +++--- include/linux/context_tracking.h | 4 ++-- kernel/context_tracking.c | 20 ++++++++++++++------ 5 files changed, 25 insertions(+), 17 deletions(-)
arch/arm/kernel/entry-header.S +4 -4
@@ -370,10 +370,10 @@ ALT_UP_B(.L1_\@) #ifdef CONFIG_CONTEXT_TRACKING .if \save stmdb sp!, {r0-r3, ip, lr} - bl context_tracking_user_exit + bl user_exit_callable ldmia sp!, {r0-r3, ip, lr} .else - bl context_tracking_user_exit + bl user_exit_callable .endif #endif .endm
@@ -382,10 +382,10 @@ ALT_UP_B(.L1_\@) #ifdef CONFIG_CONTEXT_TRACKING .if \save stmdb sp!, {r0-r3, ip, lr} - bl context_tracking_user_enter + bl user_enter_callable ldmia sp!, {r0-r3, ip, lr} .else - bl context_tracking_user_enter + bl user_enter_callable .endif #endif .endm
arch/csky/kernel/entry.S +2 -2
@@ -23,7 +23,7 @@ mfcr a0, epsr btsti a0, 31 bt 1f - jbsr context_tracking_user_exit + jbsr user_exit_callable ldw a0, (sp, LSAVE_A0) ldw a1, (sp, LSAVE_A1) ldw a2, (sp, LSAVE_A2)
@@ -160,7 +160,7 @@ ret_from_exception: cmpnei r10, 0 bt exit_work #ifdef CONFIG_CONTEXT_TRACKING - jbsr context_tracking_user_enter + jbsr user_enter_callable #endif 1: #ifdef CONFIG_PREEMPTION
arch/riscv/kernel/entry.S +3 -3
@@ -112,11 +112,11 @@ _save_context: #endif #ifdef CONFIG_CONTEXT_TRACKING - /* If previous state is in user mode, call context_tracking_user_exit. */ + /* If previous state is in user mode, call user_exit_callable(). */ li a0, SR_PP and a0, s1, a0 bnez a0, skip_context_tracking - call context_tracking_user_exit + call user_exit_callable skip_context_tracking: #endif
@@ -256,7 +256,7 @@ resume_userspace: bnez s1, work_pending #ifdef CONFIG_CONTEXT_TRACKING - call context_tracking_user_enter + call user_enter_callable #endif /* Save unwound kernel stack pointer in thread_info */
include/linux/context_tracking.h +2 -2
@@ -19,8 +19,8 @@ extern void __ct_user_exit(enum ctx_state state);extern void context_tracking_enter (enum ctx_state state);extern void context_tracking_exit (enum ctx_state state); -extern void context_tracking_user_enter (void ); -extern void context_tracking_user_exit (void ); +extern void user_enter_callable (void ); +extern void user_exit_callable (void );static inline void user_enter (void ) {
kernel/context_tracking.c +14 -6
@@ -130,15 +130,19 @@ void context_tracking_enter(enum ctx_state state)NOKPROBE_SYMBOL (context_tracking_enter);EXPORT_SYMBOL_GPL (context_tracking_enter); -/* +/** +* user_enter_callable() - Unfortunate ASM callable version of user_enter() for +* archs that didn't manage to check the context tracking +* static key from low level code. +* * FIXME: This function should be noinstr but it unsafely calls local_irq_restore(), * involving illegal RCU uses through tracing and lockdep. This must be fixed first. */ -void context_tracking_user_enter (void ) +void user_enter_callable (void ) {user_enter (); } -NOKPROBE_SYMBOL (context_tracking_user_enter); +NOKPROBE_SYMBOL (user_enter_callable);/** * __ct_user_exit - Inform the context tracking that the CPU is
@@ -196,15 +200,19 @@ void context_tracking_exit(enum ctx_state state)NOKPROBE_SYMBOL (context_tracking_exit);EXPORT_SYMBOL_GPL (context_tracking_exit); -/* +/** +* user_exit_callable() - Unfortunate ASM callable version of user_exit() for +* archs that didn't manage to check the context tracking +* static key from low level code. +* * FIXME: This function should be noinstr but it unsafely calls local_irq_save(), * involving illegal RCU uses through tracing and lockdep. This must be fixed first. */ -void context_tracking_user_exit (void ) +void user_exit_callable (void ) {user_exit (); } -NOKPROBE_SYMBOL (context_tracking_user_exit); +NOKPROBE_SYMBOL (user_exit_callable);void __init context_tracking_cpu_set (int cpu) {
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 7/21 context_tracking: Rename context_tracking_enter/exit() to ct_user_enter/exit()
In reply to Frederic Weisbeckercontext_tracking_enter() and context_tracking_exit() have confusing names that don't explain the fact they are referring to user/guest state.
Use more self-explanatory names and shrink to the new context tracking prefix instead.
--- include/linux/context_tracking.h | 13 +++++++------ kernel/context_tracking.c | 12 ++++++------ 2 files changed, 13 insertions(+), 12 deletions(-)
include/linux/context_tracking.h +7 -6
@@ -17,21 +17,22 @@ extern void context_tracking_cpu_set(int cpu);extern void __ct_user_enter (enum ctx_state state);extern void __ct_user_exit (enum ctx_state state); -extern void context_tracking_enter (enum ctx_state state); -extern void context_tracking_exit (enum ctx_state state); +extern void ct_user_enter (enum ctx_state state); +extern void ct_user_exit (enum ctx_state state); +extern void user_enter_callable (void );extern void user_exit_callable (void );static inline void user_enter (void ) {if (context_tracking_enabled ()) -context_tracking_enter (CONTEXT_USER ); +ct_user_enter (CONTEXT_USER ); }static inline void user_exit (void ) {if (context_tracking_enabled ()) -context_tracking_exit (CONTEXT_USER ); +ct_user_exit (CONTEXT_USER ); }/* Called with interrupts disabled. */
@@ -57,7 +58,7 @@ static inline enum ctx_state exception_enter(void) prev_ctx =this_cpu_read (context_tracking.state);if (prev_ctx !=CONTEXT_KERNEL ) -context_tracking_exit (prev_ctx); +ct_user_exit (prev_ctx);return prev_ctx; }
@@ -67,7 +68,7 @@ static inline void exception_exit(enum ctx_state prev_ctx)if (!IS_ENABLED (CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK ) &&context_tracking_enabled ()) {if (prev_ctx !=CONTEXT_KERNEL ) -context_tracking_enter (prev_ctx); +ct_user_enter (prev_ctx); } }
kernel/context_tracking.c +6 -6
@@ -108,7 +108,7 @@ EXPORT_SYMBOL_GPL(__ct_user_enter);* unsafe because it involves illegal RCU uses through tracing and lockdep. This * must be fixed first. */ -void context_tracking_enter (enum ctx_state state) +void ct_user_enter (enum ctx_state state) {unsigned long flags;
@@ -127,8 +127,8 @@ void context_tracking_enter(enum ctx_state state)__ct_user_enter (state);local_irq_restore (flags); } -NOKPROBE_SYMBOL (context_tracking_enter); -EXPORT_SYMBOL_GPL (context_tracking_enter); +NOKPROBE_SYMBOL (ct_user_enter); +EXPORT_SYMBOL_GPL (ct_user_enter);/** * user_enter_callable() - Unfortunate ASM callable version of user_enter() for
@@ -186,7 +186,7 @@ EXPORT_SYMBOL_GPL(__ct_user_exit);* unsafe because it involves illegal RCU uses through tracing and lockdep. This * must be fixed first. */ -void context_tracking_exit (enum ctx_state state) +void ct_user_exit (enum ctx_state state) {unsigned long flags;
@@ -197,8 +197,8 @@ void context_tracking_exit(enum ctx_state state)__ct_user_exit (state);local_irq_restore (flags); } -NOKPROBE_SYMBOL (context_tracking_exit); -EXPORT_SYMBOL_GPL (context_tracking_exit); +NOKPROBE_SYMBOL (ct_user_exit); +EXPORT_SYMBOL_GPL (ct_user_exit);/** * user_exit_callable() - Unfortunate ASM callable version of user_exit() for
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 8/21 context_tracking: Rename context_tracking_cpu_set() to ct_cpu_track_user()
In reply to Frederic Weisbeckercontext_tracking_cpu_set() is called in order to tell a CPU to track user/kernel transitions. Since context tracking is going to expand in to also track transitions from/to idle/IRQ/NMIs, the scope of this function name becomes too broad and needs to be made more specific. Also shorten the prefix to align with the new namespace.
--- include/linux/context_tracking.h | 2 +- kernel/context_tracking.c | 4 ++-- kernel/time/tick-sched.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
include/linux/context_tracking.h +1 -1
@@ -11,7 +11,7 @@#ifdef CONFIG_CONTEXT_TRACKING -extern void context_tracking_cpu_set (int cpu); +extern void ct_cpu_track_user (int cpu);/* Called with interrupts disabled. */ extern void __ct_user_enter (enum ctx_state state);
kernel/context_tracking.c +2 -2
@@ -214,7 +214,7 @@ void user_exit_callable(void) }NOKPROBE_SYMBOL (user_exit_callable); -void __init context_tracking_cpu_set (int cpu) +void __init ct_cpu_track_user (int cpu) {static __initdata bool initialized = false;
@@ -244,6 +244,6 @@ void __init context_tracking_init(void)int cpu;for_each_possible_cpu (cpu) - context_tracking_cpu_set(cpu); + ct_cpu_track_user(cpu); }#endif
kernel/time/tick-sched.c +1 -1
@@ -571,7 +571,7 @@ void __init tick_nohz_init(void) }for_each_cpu (cpu, tick_nohz_full_mask) -context_tracking_cpu_set (cpu); +ct_cpu_track_user (cpu); ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "kernel/nohz:predown", NULL,
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 9/21 context_tracking: Split user tracking Kconfig
In reply to Frederic WeisbeckerContext tracking is going to be used not only to track user transitions but also idle/IRQs/NMIs. The user tracking part will then become a separate feature. Prepare Kconfig for that.
--- .../time/context-tracking/arch-support.txt | 6 ++--- arch/Kconfig | 4 ++-- arch/arm/Kconfig | 2 +- arch/arm/kernel/entry-common.S | 4 ++-- arch/arm/kernel/entry-header.S | 4 ++-- arch/arm64/Kconfig | 2 +- arch/csky/Kconfig | 2 +- arch/csky/kernel/entry.S | 4 ++-- arch/mips/Kconfig | 2 +- arch/powerpc/Kconfig | 2 +- arch/powerpc/include/asm/context_tracking.h | 2 +- arch/riscv/Kconfig | 2 +- arch/riscv/kernel/entry.S | 6 ++--- arch/sparc/Kconfig | 2 +- arch/sparc/kernel/rtrap_64.S | 2 +- arch/x86/Kconfig | 4 ++-- include/linux/context_tracking.h | 12 +++++----- include/linux/context_tracking_state.h | 4 ++-- init/Kconfig | 4 ++-- kernel/context_tracking.c | 6 ++++- kernel/sched/core.c | 2 +- kernel/time/Kconfig | 22 +++++++++++-------- 22 files changed, 54 insertions(+), 46 deletions(-)
Documentation/features/time/context-tracking/arch-support.txt +3 -3
@@ -1,7 +1,7 @@ # -# Feature name: context-tracking -# Kconfig: HAVE_CONTEXT_TRACKING -# description: arch supports context tracking for NO_HZ_FULL +# Feature name: user-context-tracking +# Kconfig: HAVE_CONTEXT_TRACKING_USER +# description: arch supports user context tracking for NO_HZ_FULL # ----------------------- | arch |status|
arch/Kconfig +2 -2
@@ -769,7 +769,7 @@ config HAVE_ARCH_WITHIN_STACK_FRAMES and similar) by implementing an inline arch_within_stack_frames(), which is used by CONFIG_HARDENED_USERCOPY. -config HAVE_CONTEXT_TRACKING +config HAVE_CONTEXT_TRACKING_USER bool help Provide kernel/user boundaries probes necessary for subsystems
@@ -780,7 +780,7 @@ config HAVE_CONTEXT_TRACKING protected inside rcu_irq_enter/rcu_irq_exit() but preemption or signal handling on irq exit still need to be protected. -config HAVE_CONTEXT_TRACKING_OFFSTACK +config HAVE_CONTEXT_TRACKING_USER_OFFSTACK bool help Architecture neither relies on exception_enter()/exception_exit()
arch/arm/Kconfig +1 -1
@@ -84,7 +84,7 @@ config ARM select HAVE_ARCH_TRANSPARENT_HUGEPAGE if ARM_LPAE select HAVE_ARM_SMCCC if CPU_V7 select HAVE_EBPF_JIT if !CPU_ENDIAN_BE32 - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_C_RECORDMCOUNT select HAVE_BUILDTIME_MCOUNT_SORT select HAVE_DEBUG_KMEMLEAK if !XIP_KERNEL
arch/arm/kernel/entry-common.S +2 -2
@@ -28,7 +28,7 @@ #include "entry-header.S" saved_psr .req r8 -#if defined(CONFIG_TRACE_IRQFLAGS) || defined(CONFIG_CONTEXT_TRACKING) +#if defined(CONFIG_TRACE_IRQFLAGS) || defined(CONFIG_CONTEXT_TRACKING_USER) saved_pc .req r9 #define TRACE(x...) x #else
@@ -38,7 +38,7 @@ saved_pc .req lr .section .entry.text,"ax",%progbits .align 5 -#if !(IS_ENABLED(CONFIG_TRACE_IRQFLAGS) || IS_ENABLED(CONFIG_CONTEXT_TRACKING) || \ +#if !(IS_ENABLED(CONFIG_TRACE_IRQFLAGS) || IS_ENABLED(CONFIG_CONTEXT_TRACKING_USER) || \ IS_ENABLED(CONFIG_DEBUG_RSEQ)) /* * This is the fast syscall return path. We do as little as possible here,
arch/arm/kernel/entry-header.S +2 -2
@@ -367,7 +367,7 @@ ALT_UP_B(.L1_\@) * between user and kernel mode. */ .macro ct_user_exit, save = 1 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER .if \save stmdb sp!, {r0-r3, ip, lr} bl user_exit_callable
@@ -379,7 +379,7 @@ ALT_UP_B(.L1_\@) .endm .macro ct_user_enter, save = 1 -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER .if \save stmdb sp!, {r0-r3, ip, lr} bl user_enter_callable
arch/arm64/Kconfig +1 -1
@@ -171,7 +171,7 @@ config ARM64 select HAVE_C_RECORDMCOUNT select HAVE_CMPXCHG_DOUBLE select HAVE_CMPXCHG_LOCAL - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_DEBUG_KMEMLEAK select HAVE_DMA_CONTIGUOUS select HAVE_DYNAMIC_FTRACE
arch/csky/Kconfig +1 -1
@@ -42,7 +42,7 @@ config CSKY select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_SECCOMP_FILTER - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_VIRT_CPU_ACCOUNTING_GEN select HAVE_DEBUG_BUGVERBOSE select HAVE_DEBUG_KMEMLEAK
arch/csky/kernel/entry.S +2 -2
@@ -19,7 +19,7 @@ .endm .macro context_tracking -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER mfcr a0, epsr btsti a0, 31 bt 1f
@@ -159,7 +159,7 @@ ret_from_exception: and r10, r9 cmpnei r10, 0 bt exit_work -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER jbsr user_enter_callable #endif 1:
arch/mips/Kconfig +1 -1
@@ -56,7 +56,7 @@ config MIPS select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE if CPU_SUPPORTS_HUGEPAGES select HAVE_ASM_MODVERSIONS - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_TIF_NOHZ select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK
arch/powerpc/Kconfig +1 -1
@@ -197,7 +197,7 @@ config PPC select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ASM_MODVERSIONS - select HAVE_CONTEXT_TRACKING if PPC64 + select HAVE_CONTEXT_TRACKING_USER if PPC64 select HAVE_C_RECORDMCOUNT select HAVE_DEBUG_KMEMLEAK select HAVE_DEBUG_STACKOVERFLOW
arch/powerpc/include/asm/context_tracking.h +1 -1
@@ -2,7 +2,7 @@#ifndef _ASM_POWERPC_CONTEXT_TRACKING_H#define _ASM_POWERPC_CONTEXT_TRACKING_H -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER #define SCHEDULE_USER bl schedule_user#else #define SCHEDULE_USER bl schedule
arch/riscv/Kconfig +1 -1
@@ -83,7 +83,7 @@ config RISCV select HAVE_ARCH_THREAD_STRUCT_WHITELIST select HAVE_ARCH_VMAP_STACK if MMU && 64BIT select HAVE_ASM_MODVERSIONS - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_DEBUG_KMEMLEAK select HAVE_DMA_CONTIGUOUS if MMU select HAVE_EBPF_JIT if MMU
arch/riscv/kernel/entry.S +3 -3
@@ -111,7 +111,7 @@ _save_context: call __trace_hardirqs_off #endif -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER /* If previous state is in user mode, call user_exit_callable(). */ li a0, SR_PP and a0, s1, a0
@@ -176,7 +176,7 @@ handle_syscall: */ csrs CSR_STATUS, SR_IE #endif -#if defined(CONFIG_TRACE_IRQFLAGS) || defined(CONFIG_CONTEXT_TRACKING) +#if defined(CONFIG_TRACE_IRQFLAGS) || defined(CONFIG_CONTEXT_TRACKING_USER) /* Recover a0 - a7 for system calls */ REG_L a0, PT_A0(sp) REG_L a1, PT_A1(sp)
@@ -255,7 +255,7 @@ resume_userspace: andi s1, s0, _TIF_WORK_MASK bnez s1, work_pending -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER call user_enter_callable #endif
arch/sparc/Kconfig +1 -1
@@ -71,7 +71,7 @@ config SPARC64 select HAVE_DYNAMIC_FTRACE select HAVE_FTRACE_MCOUNT_RECORD select HAVE_SYSCALL_TRACEPOINTS - select HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER select HAVE_TIF_NOHZ select HAVE_DEBUG_KMEMLEAK select IOMMU_HELPER
arch/sparc/kernel/rtrap_64.S +1 -1
@@ -15,7 +15,7 @@ #include <asm/visasm.h> #include <asm/processor.h> -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER # define SCHEDULE_USER schedule_user #else # define SCHEDULE_USER schedule
arch/x86/Kconfig +2 -2
@@ -185,8 +185,8 @@ config X86 select HAVE_ASM_MODVERSIONS select HAVE_CMPXCHG_DOUBLE select HAVE_CMPXCHG_LOCAL - select HAVE_CONTEXT_TRACKING if X86_64 - select HAVE_CONTEXT_TRACKING_OFFSTACK if HAVE_CONTEXT_TRACKING + select HAVE_CONTEXT_TRACKING_USER if X86_64 + select HAVE_CONTEXT_TRACKING_USER_OFFSTACK if HAVE_CONTEXT_TRACKING_USER select HAVE_C_RECORDMCOUNT select HAVE_OBJTOOL_MCOUNT if STACK_VALIDATION select HAVE_BUILDTIME_MCOUNT_SORT
include/linux/context_tracking.h +6 -6
@@ -10,7 +10,7 @@#include <asm/ptrace.h> -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER extern void ct_cpu_track_user (int cpu);/* Called with interrupts disabled. */
@@ -52,7 +52,7 @@ static inline enum ctx_state exception_enter(void) {enum ctx_state prev_ctx; -if (IS_ENABLED (CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK ) || +if (IS_ENABLED (CONFIG_HAVE_CONTEXT_TRACKING_USER_OFFSTACK ) || !context_tracking_enabled ())return 0 ;
@@ -65,7 +65,7 @@ static inline enum ctx_state exception_enter(void)static inline void exception_exit (enum ctx_state prev_ctx) { -if (!IS_ENABLED (CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK ) && +if (!IS_ENABLED (CONFIG_HAVE_CONTEXT_TRACKING_USER_OFFSTACK ) &&context_tracking_enabled ()) {if (prev_ctx !=CONTEXT_KERNEL )ct_user_enter (prev_ctx);
@@ -109,14 +109,14 @@ static inline enum ctx_state ct_state(void) { return CONTEXT_DISABLED; } static __always_inline bool context_tracking_guest_enter(void) { return false; }static inline void context_tracking_guest_exit (void ) { } -#endif /* !CONFIG_CONTEXT_TRACKING */ +#endif /* !CONFIG_CONTEXT_TRACKING_USER */ #define CT_WARN_ON (cond) WARN_ON(context_tracking_enabled() && (cond)) -#ifdef CONFIG_CONTEXT_TRACKING_FORCE +#ifdef CONFIG_CONTEXT_TRACKING_USER_FORCE extern void context_tracking_init (void );#else static inline void context_tracking_init (void ) { } -#endif /* CONFIG_CONTEXT_TRACKING_FORCE */ +#endif /* CONFIG_CONTEXT_TRACKING_USER_FORCE */ #endif
include/linux/context_tracking_state.h +2 -2
@@ -22,7 +22,7 @@ struct context_tracking { } state; }; -#ifdef CONFIG_CONTEXT_TRACKING +#ifdef CONFIG_CONTEXT_TRACKING_USER extern struct static_key_false context_tracking_key; DECLARE_PER_CPU(struct context_tracking, context_tracking);
@@ -45,6 +45,6 @@ static inline bool context_tracking_enabled_this_cpu(void)static inline bool context_tracking_enabled (void ) {return false; }static inline bool context_tracking_enabled_cpu (int cpu) {return false; }static inline bool context_tracking_enabled_this_cpu (void ) {return false; } -#endif /* CONFIG_CONTEXT_TRACKING */ +#endif /* CONFIG_CONTEXT_TRACKING_USER */ #endif
init/Kconfig +2 -2
@@ -502,11 +502,11 @@ config VIRT_CPU_ACCOUNTING_NATIVE config VIRT_CPU_ACCOUNTING_GEN bool "Full dynticks CPU time accounting" - depends on HAVE_CONTEXT_TRACKING + depends on HAVE_CONTEXT_TRACKING_USER depends on HAVE_VIRT_CPU_ACCOUNTING_GEN depends on GENERIC_CLOCKEVENTS select VIRT_CPU_ACCOUNTING - select CONTEXT_TRACKING + select CONTEXT_TRACKING_USER help Select this option to enable task and CPU time accounting on full dynticks systems. This accounting is implemented by watching every
kernel/context_tracking.c +5 -1
@@ -22,6 +22,8 @@#include <linux/export.h> #include <linux/kprobes.h> +#ifdef CONFIG_CONTEXT_TRACKING_USER +#define CREATE_TRACE_POINTS #include <trace/events/context_tracking.h>
@@ -238,7 +240,7 @@ void __init ct_cpu_track_user(int cpu) initialized = true; } -#ifdef CONFIG_CONTEXT_TRACKING_FORCE +#ifdef CONFIG_CONTEXT_TRACKING_USER_FORCE void __init context_tracking_init (void ) {int cpu;
@@ -247,3 +249,5 @@ void __init context_tracking_init(void)ct_cpu_track_user (cpu); }#endif + +#endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */
kernel/sched/core.c +1 -1
@@ -6483,7 +6483,7 @@ void __sched schedule_idle(void) }while (need_resched ()); } -#if defined(CONFIG_CONTEXT_TRACKING ) && !defined(CONFIG_HAVE_CONTEXT_TRACKING_OFFSTACK ) +#if defined(CONFIG_CONTEXT_TRACKING_USER ) && !defined(CONFIG_HAVE_CONTEXT_TRACKING_USER_OFFSTACK ) asmlinkage __visible void __sched schedule_user(void) {/*
kernel/time/Kconfig +13 -9
@@ -111,7 +111,7 @@ config NO_HZ_FULL # NO_HZ_COMMON dependency # We need at least one periodic CPU for timekeeping depends on SMP - depends on HAVE_CONTEXT_TRACKING + depends on HAVE_CONTEXT_TRACKING_USER # VIRT_CPU_ACCOUNTING_GEN dependency depends on HAVE_VIRT_CPU_ACCOUNTING_GEN select NO_HZ_COMMON
@@ -140,28 +140,32 @@ endchoice config CONTEXT_TRACKING bool -config CONTEXT_TRACKING_FORCE - bool "Force context tracking" - depends on CONTEXT_TRACKING +config CONTEXT_TRACKING_USER + select CONTEXT_TRACKING + bool + +config CONTEXT_TRACKING_USER_FORCE + bool "Force user context tracking" + depends on CONTEXT_TRACKING_USER default y if !NO_HZ_FULL help The major pre-requirement for full dynticks to work is to - support the context tracking subsystem. But there are also + support the user context tracking subsystem. But there are also other dependencies to provide in order to make the full dynticks working. This option stands for testing when an arch implements the - context tracking backend but doesn't yet fulfill all the + user context tracking backend but doesn't yet fulfill all the requirements to make the full dynticks feature working. Without the full dynticks, there is no way to test the support - for context tracking and the subsystems that rely on it: RCU + for user context tracking and the subsystems that rely on it: RCU userspace extended quiescent state and tickless cputime accounting. This option copes with the absence of the full - dynticks subsystem by forcing the context tracking on all + dynticks subsystem by forcing the user context tracking on all CPUs in the system. Say Y only if you're working on the development of an - architecture backend for the context tracking. + architecture backend for the user context tracking. Say N otherwise, this option brings an overhead that you don't want in production.
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 10/21 context_tracking: Take idle eqs entrypoints over RCU
In reply to Frederic WeisbeckerThe RCU dynticks counter is going to be merged into the context tracking subsystem. Start with moving the idle extended quiescent states entrypoints to context tracking. For now those are dumb redirections to existing RCU calls.
--- Documentation/RCU/stallwarn.rst | 4 ++-- arch/arm/mach-imx/cpuidle-imx6q.c | 5 +++-- drivers/acpi/processor_idle.c | 5 +++-- drivers/cpuidle/cpuidle.c | 9 +++++---- include/linux/context_tracking.h | 8 ++++++++ include/linux/rcupdate.h | 2 +- kernel/context_tracking.c | 12 ++++++++++++ kernel/locking/lockdep.c | 2 +- kernel/rcu/Kconfig | 2 ++ kernel/rcu/tree.c | 2 -- kernel/rcu/update.c | 2 +- kernel/sched/idle.c | 10 +++++----- kernel/sched/sched.h | 1 + 13 files changed, 44 insertions(+), 20 deletions(-)
Documentation/RCU/stallwarn.rst +2 -2
@@ -97,8 +97,8 @@ warnings: which will include additional debugging information. - A low-level kernel issue that either fails to invoke one of the - variants of rcu_user_enter(), rcu_user_exit(), rcu_idle_enter(), - rcu_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one + variants of rcu_user_enter(), rcu_user_exit(), ct_idle_enter(), + ct_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one hand, or that invokes one of them too many times on the other. Historically, the most frequent issue has been an omission of either irq_enter() or irq_exit(), which in turn invoke
arch/arm/mach-imx/cpuidle-imx6q.c +3 -2
@@ -3,6 +3,7 @@* Copyright (C) 2012 Freescale Semiconductor, Inc. */ +#include <linux/context_tracking.h> #include <linux/cpuidle.h> #include <linux/module.h> #include <asm/cpuidle.h>
@@ -24,9 +25,9 @@ static int imx6q_enter_wait(struct cpuidle_device *dev,imx6_set_lpm (WAIT_UNCLOCKED );raw_spin_unlock (&cpuidle_lock); -rcu_idle_enter (); +ct_idle_enter ();cpu_do_idle (); -rcu_idle_exit (); +ct_idle_exit ();raw_spin_lock (&cpuidle_lock);if (num_idle_cpus-- ==num_online_cpus ())
drivers/acpi/processor_idle.c +3 -2
@@ -22,6 +22,7 @@#include <linux/cpu.h> #include <linux/minmax.h> #include <acpi/processor.h> +#include <linux/context_tracking.h> /* * Include the apic definitions for x86 to have the APIC timer related defines
@@ -648,11 +649,11 @@ static int acpi_idle_enter_bm(struct cpuidle_driver *drv,raw_spin_unlock (&c3_lock); } -rcu_idle_enter (); +ct_idle_enter ();acpi_idle_do_entry (cx); -rcu_idle_exit (); +ct_idle_exit ();/* Re-enable bus master arbitration */ if (dis_bm) {
drivers/cpuidle/cpuidle.c +5 -4
@@ -23,6 +23,7 @@#include <linux/suspend.h> #include <linux/tick.h> #include <linux/mmu_context.h> +#include <linux/context_tracking.h> #include <trace/events/power.h> #include "cpuidle.h"
@@ -150,12 +151,12 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,*/ stop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -rcu_idle_enter (); +ct_idle_enter (); target_state->enter_s2idle (dev, drv, index);if (WARN_ON_ONCE (!irqs_disabled ()))local_irq_disable ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -rcu_idle_exit (); +ct_idle_exit ();tick_unfreeze ();start_critical_timings ();
@@ -233,10 +234,10 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,stop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -rcu_idle_enter (); +ct_idle_enter (); entered_state = target_state->enter (dev, drv, index);if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -rcu_idle_exit (); +ct_idle_exit ();start_critical_timings ();sched_clock_idle_wakeup_event ();
include/linux/context_tracking.h +8 -0
@@ -119,4 +119,12 @@ extern void context_tracking_init(void);static inline void context_tracking_init (void ) { }#endif /* CONFIG_CONTEXT_TRACKING_USER_FORCE */ +#ifdef CONFIG_CONTEXT_TRACKING +extern void ct_idle_enter (void ); +extern void ct_idle_exit (void ); +#else +static inline void ct_idle_enter (void ) { } +static inline void ct_idle_exit (void ) { } +#endif /* !CONFIG_CONTEXT_TRACKING */ +#endif
include/linux/rcupdate.h +1 -1
@@ -129,7 +129,7 @@ static inline void rcu_nocb_flush_deferred_wakeup(void) { }* @a: Code that RCU needs to pay attention to. * * RCU read-side critical sections are forbidden in the inner idle loop, -* that is, between the rcu_idle_enter() and the rcu_idle_exit() -- RCU +* that is, between the ct_idle_enter() and the ct_idle_exit() -- RCU * will happily ignore any such read-side critical sections. However, * things like powertop need tracepoints in the inner idle loop. *
kernel/context_tracking.c +12 -0
@@ -251,3 +251,15 @@ void __init context_tracking_init(void)#endif #endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ + +noinstr void ct_idle_enter(void) +{ +rcu_idle_enter (); +} +EXPORT_SYMBOL_GPL (ct_idle_enter); + +void ct_idle_exit (void ) +{ +rcu_idle_exit (); +} +EXPORT_SYMBOL_GPL (ct_idle_exit);
kernel/locking/lockdep.c +1 -1
@@ -6546,7 +6546,7 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)/* * If a CPU is in the RCU-free window in idle (ie: in the section -* between rcu_idle_enter() and rcu_idle_exit(), then RCU +* between ct_idle_enter() and ct_idle_exit(), then RCU * considers that CPU to be in an "extended quiescent state", * which means that RCU will be completely ignoring that CPU. * Therefore, rcu_read_lock() and friends have absolutely no
kernel/rcu/Kconfig +2 -0
@@ -8,6 +8,8 @@ menu "RCU Subsystem" config TREE_RCU bool default y if SMP + # Dynticks-idle tracking + select CONTEXT_TRACKING help This option selects the RCU implementation that is designed for very large SMP system with hundreds or
kernel/rcu/tree.c +0 -2
@@ -664,7 +664,6 @@ noinstr void rcu_idle_enter(void)lockdep_assert_irqs_disabled ();rcu_eqs_enter (false); } -EXPORT_SYMBOL_GPL (rcu_idle_enter);#ifdef CONFIG_NO_HZ_FULL
@@ -912,7 +911,6 @@ void rcu_idle_exit(void)rcu_eqs_exit (false);local_irq_restore (flags); } -EXPORT_SYMBOL_GPL (rcu_idle_exit);#ifdef CONFIG_NO_HZ_FULL /**
kernel/rcu/update.c +1 -1
@@ -85,7 +85,7 @@ module_param(rcu_normal_after_boot, int, 0444);* and while lockdep is disabled. * * Note that if the CPU is in the idle loop from an RCU point of view (ie: -* that we are in the section between rcu_idle_enter() and rcu_idle_exit()) +* that we are in the section between ct_idle_enter() and ct_idle_exit()) * then rcu_read_lock_held() sets ``*ret`` to false even if the CPU did an * rcu_read_lock(). The reason for this is that RCU ignores CPUs that are * in such a section, considering these as in extended quiescent state,
kernel/sched/idle.c +5 -5
@@ -53,14 +53,14 @@ static noinline int __cpuidle cpu_idle_poll(void) {trace_cpu_idle (0 ,smp_processor_id ());stop_critical_timings (); -rcu_idle_enter (); +ct_idle_enter ();local_irq_enable ();while (!tif_need_resched () && (cpu_idle_force_poll ||tick_check_broadcast_expired ()))cpu_relax (); -rcu_idle_exit (); +ct_idle_exit ();start_critical_timings ();trace_cpu_idle (PWR_EVENT_EXIT ,smp_processor_id ());
@@ -98,12 +98,12 @@ void __cpuidle default_idle_call(void)* * Trace IRQs enable here, then switch off RCU, and have * arch_cpu_idle() use raw_local_irq_enable(). Note that -* rcu_idle_enter() relies on lockdep IRQ state, so switch that +* ct_idle_enter() relies on lockdep IRQ state, so switch that * last -- this is very similar to the entry code. */ trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (_THIS_IP_); -rcu_idle_enter (); +ct_idle_enter ();lockdep_hardirqs_on (_THIS_IP_);arch_cpu_idle ();
@@ -116,7 +116,7 @@ void __cpuidle default_idle_call(void)*/ raw_local_irq_disable ();lockdep_hardirqs_off (_THIS_IP_); -rcu_idle_exit (); +ct_idle_exit ();lockdep_hardirqs_on (_THIS_IP_);raw_local_irq_enable ();
kernel/sched/sched.h +1 -0
@@ -27,6 +27,7 @@#include <linux/capability.h> #include <linux/cgroup_api.h> #include <linux/cgroup.h> +#include <linux/context_tracking.h> #include <linux/cpufreq.h> #include <linux/cpumask_api.h> #include <linux/ctype.h>
Signature · 1 line
2.25.1
Mark Rutland
reviewerRe: [PATCH 10/21] context_tracking: Take idle eqs entrypoints over RCU
Hi Frederic,
Quoted 4 lines · Frederic Weisbecker
The RCU dynticks counter is going to be merged into the context tracking subsystem. Start with moving the idle extended quiescent states entrypoints to context tracking. For now those are dumb redirections to existing RCU calls.
I was a bit confused looking at this, because that redirection only exists for CONFIG_CONTEXT_TRACKING, and is empty otherwise.
I see this patch makes TREE_RCU select CONTEXT_TRACKING, which means that works. Since that also means building the rest of the context tracking code, I think it'd be worth mentioning that in the commit message.
Do all architectures which can use TREE_RCU today already support context tracking? If not, do those work by default?
Thanks,
Mark.
Quoted 307 lines
Signed-off-by Frederic Weisbecker <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Neeraj Upadhyay <[email protected]>Cc Uladzislau Rezki <[email protected]>Cc Joel Fernandes <[email protected]>Cc Boqun Feng <[email protected]>Cc Nicolas Saenz Julienne <[email protected]>Cc Marcelo Tosatti <[email protected]>Cc Xiongfeng Wang <[email protected]>Cc Yu Liao<[email protected]>Cc Phil Auld <[email protected]>Cc Paul Gortmaker<[email protected]>Cc Alex Belits <[email protected]>--- Documentation/RCU/stallwarn.rst | 4 ++-- arch/arm/mach-imx/cpuidle-imx6q.c | 5 +++-- drivers/acpi/processor_idle.c | 5 +++-- drivers/cpuidle/cpuidle.c | 9 +++++---- include/linux/context_tracking.h | 8 ++++++++ include/linux/rcupdate.h | 2 +- kernel/context_tracking.c | 12 ++++++++++++ kernel/locking/lockdep.c | 2 +- kernel/rcu/Kconfig | 2 ++ kernel/rcu/tree.c | 2 -- kernel/rcu/update.c | 2 +- kernel/sched/idle.c | 10 +++++----- kernel/sched/sched.h | 1 + 13 files changed, 44 insertions(+), 20 deletions(-)Documentation/RCU/stallwarn.rst +2
-2@@ -97,8 +97,8 @@ warnings: which will include additional debugging information. - A low-level kernel issue that either fails to invoke one of the - variants of rcu_user_enter(), rcu_user_exit(), rcu_idle_enter(), - rcu_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one + variants of rcu_user_enter(), rcu_user_exit(), ct_idle_enter(), + ct_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one hand, or that invokes one of them too many times on the other. Historically, the most frequent issue has been an omission of either irq_enter() or irq_exit(), which in turn invokearch/arm/mach-imx/cpuidle-imx6q.c +3
-2@@ -3,6 +3,7 @@* Copyright (C) 2012 Freescale Semiconductor, Inc. */ +#include <linux/context_tracking.h> #include <linux/cpuidle.h> #include <linux/module.h> #include <asm/cpuidle.h> @@ -24,9 +25,9 @@ static int imx6q_enter_wait(struct cpuidle_device *dev,imx6_set_lpm (WAIT_UNCLOCKED );raw_spin_unlock (&cpuidle_lock); -rcu_idle_enter (); +ct_idle_enter ();cpu_do_idle (); -rcu_idle_exit (); +ct_idle_exit ();raw_spin_lock (&cpuidle_lock);if (num_idle_cpus-- ==num_online_cpus ())drivers/acpi/processor_idle.c +3
-2@@ -22,6 +22,7 @@#include <linux/cpu.h> #include <linux/minmax.h> #include <acpi/processor.h> +#include <linux/context_tracking.h> /* * Include the apic definitions for x86 to have the APIC timer related defines @@ -648,11 +649,11 @@ static int acpi_idle_enter_bm(struct cpuidle_driver *drv,raw_spin_unlock (&c3_lock); } -rcu_idle_enter (); +ct_idle_enter ();acpi_idle_do_entry (cx); -rcu_idle_exit (); +ct_idle_exit ();/* Re-enable bus master arbitration */ if (dis_bm) {drivers/cpuidle/cpuidle.c +5
-4@@ -23,6 +23,7 @@#include <linux/suspend.h> #include <linux/tick.h> #include <linux/mmu_context.h> +#include <linux/context_tracking.h> #include <trace/events/power.h> #include "cpuidle.h" @@ -150,12 +151,12 @@ static void enter_s2idle_proper(struct cpuidle_driver *drv,*/ stop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -rcu_idle_enter (); +ct_idle_enter (); target_state->enter_s2idle (dev, drv, index);if (WARN_ON_ONCE (!irqs_disabled ()))local_irq_disable ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -rcu_idle_exit (); +ct_idle_exit ();tick_unfreeze ();start_critical_timings ();@@ -233,10 +234,10 @@ int cpuidle_enter_state(struct cpuidle_device *dev, struct cpuidle_driver *drv,stop_critical_timings ();if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -rcu_idle_enter (); +ct_idle_enter (); entered_state = target_state->enter (dev, drv, index);if (!(target_state->flags &CPUIDLE_FLAG_RCU_IDLE )) -rcu_idle_exit (); +ct_idle_exit ();start_critical_timings ();sched_clock_idle_wakeup_event ();include/linux/context_tracking.h +8
-0@@ -119,4 +119,12 @@ extern void context_tracking_init(void);static inline void context_tracking_init (void ) { }#endif /* CONFIG_CONTEXT_TRACKING_USER_FORCE */ +#ifdef CONFIG_CONTEXT_TRACKING +extern void ct_idle_enter (void ); +extern void ct_idle_exit (void ); +#else +static inline void ct_idle_enter (void ) { } +static inline void ct_idle_exit (void ) { } +#endif /* !CONFIG_CONTEXT_TRACKING */ +#endif include/linux/rcupdate.h +1
-1@@ -129,7 +129,7 @@ static inline void rcu_nocb_flush_deferred_wakeup(void) { }* @a: Code that RCU needs to pay attention to. * * RCU read-side critical sections are forbidden in the inner idle loop, -* that is, between the rcu_idle_enter() and the rcu_idle_exit() -- RCU +* that is, between the ct_idle_enter() and the ct_idle_exit() -- RCU * will happily ignore any such read-side critical sections. However, * things like powertop need tracepoints in the inner idle loop. * kernel/context_tracking.c +12
-0@@ -251,3 +251,15 @@ void __init context_tracking_init(void)#endif #endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ + +noinstr void ct_idle_enter(void) +{ +rcu_idle_enter (); +} +EXPORT_SYMBOL_GPL (ct_idle_enter); + +void ct_idle_exit (void ) +{ +rcu_idle_exit (); +} +EXPORT_SYMBOL_GPL (ct_idle_exit);kernel/locking/lockdep.c +1
-1@@ -6546,7 +6546,7 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s)/* * If a CPU is in the RCU-free window in idle (ie: in the section -* between rcu_idle_enter() and rcu_idle_exit(), then RCU +* between ct_idle_enter() and ct_idle_exit(), then RCU * considers that CPU to be in an "extended quiescent state", * which means that RCU will be completely ignoring that CPU. * Therefore, rcu_read_lock() and friends have absolutely no kernel/rcu/Kconfig +2
-0@@ -8,6 +8,8 @@ menu "RCU Subsystem" config TREE_RCU bool default y if SMP + # Dynticks-idle tracking + select CONTEXT_TRACKING help This option selects the RCU implementation that is designed for very large SMP system with hundreds orkernel/rcu/tree.c +0
-2@@ -664,7 +664,6 @@ noinstr void rcu_idle_enter(void)lockdep_assert_irqs_disabled ();rcu_eqs_enter (false); } -EXPORT_SYMBOL_GPL (rcu_idle_enter);#ifdef CONFIG_NO_HZ_FULL @@ -912,7 +911,6 @@ void rcu_idle_exit(void)rcu_eqs_exit (false);local_irq_restore (flags); } -EXPORT_SYMBOL_GPL (rcu_idle_exit);#ifdef CONFIG_NO_HZ_FULL /** kernel/rcu/update.c +1
-1@@ -85,7 +85,7 @@ module_param(rcu_normal_after_boot, int, 0444);* and while lockdep is disabled. * * Note that if the CPU is in the idle loop from an RCU point of view (ie: -* that we are in the section between rcu_idle_enter() and rcu_idle_exit()) +* that we are in the section between ct_idle_enter() and ct_idle_exit()) * then rcu_read_lock_held() sets ``*ret`` to false even if the CPU did an * rcu_read_lock(). The reason for this is that RCU ignores CPUs that are * in such a section, considering these as in extended quiescent state, kernel/sched/idle.c +5
-5@@ -53,14 +53,14 @@ static noinline int __cpuidle cpu_idle_poll(void) {trace_cpu_idle (0 ,smp_processor_id ());stop_critical_timings (); -rcu_idle_enter (); +ct_idle_enter ();local_irq_enable ();while (!tif_need_resched () && (cpu_idle_force_poll ||tick_check_broadcast_expired ()))cpu_relax (); -rcu_idle_exit (); +ct_idle_exit ();start_critical_timings ();trace_cpu_idle (PWR_EVENT_EXIT ,smp_processor_id ());@@ -98,12 +98,12 @@ void __cpuidle default_idle_call(void)* * Trace IRQs enable here, then switch off RCU, and have * arch_cpu_idle() use raw_local_irq_enable(). Note that -* rcu_idle_enter() relies on lockdep IRQ state, so switch that +* ct_idle_enter() relies on lockdep IRQ state, so switch that * last -- this is very similar to the entry code. */ trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (_THIS_IP_); -rcu_idle_enter (); +ct_idle_enter ();lockdep_hardirqs_on (_THIS_IP_);arch_cpu_idle ();@@ -116,7 +116,7 @@ void __cpuidle default_idle_call(void)*/ raw_local_irq_disable ();lockdep_hardirqs_off (_THIS_IP_); -rcu_idle_exit (); +ct_idle_exit ();lockdep_hardirqs_on (_THIS_IP_);raw_local_irq_enable ();kernel/sched/sched.h +1
-0@@ -27,6 +27,7 @@#include <linux/capability.h> #include <linux/cgroup_api.h> #include <linux/cgroup.h> +#include <linux/context_tracking.h> #include <linux/cpufreq.h> #include <linux/cpumask_api.h> #include <linux/ctype.h> Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerRe: [PATCH 10/21] context_tracking: Take idle eqs entrypoints over RCU
Quoted 8 lines · Mark Rutland
Hi Frederic,
Quoted 4 lines · Frederic Weisbecker
The RCU dynticks counter is going to be merged into the context tracking subsystem. Start with moving the idle extended quiescent states entrypoints to context tracking. For now those are dumb redirections to existing RCU calls.
Quoted 6 lines
I was a bit confused looking at this, because that redirection only exists for
CONFIG_CONTEXT_TRACKING, and is empty otherwise.I see this patch makes
TREE_RCUselectCONTEXT_TRACKING, which means that works. Since that also means building the rest of the context tracking code, I think it'd be worth mentioning that in the commit message.
Not exactly, it only builds a minimal version of CONTEXT_TRACKING, which maps to roughly the vanilla RCU-idle code.
The user tracking part (which is the vanilla CONTEXT_TRACKING) now becomes CONFIG_CONTEXT_TRACKING_USER
Quoted 2 lines
Do all architectures which can use
TREE_RCUtoday already support context tracking? If not, do those work by default?
No need to, the old context tracking becomes CONFIG_CONTEXT_TRACKING_USER which is not needed by TREE_RCU.
Ok you're right, this is all very confusing :o)
Let's clarify this, I'm going to do the following on the next version:
* The current CONFIG_CONTEXT_TRACKING becomes CONFIG_CONTEXT_TRACKING_USER. This part needs support from the architecture.
* The RCU-idle functions that have been moved to context tracking will become CONFIG_CONTEXT_TRACKING_IDLE. This part doesn't need support from the architecture and will be set by TREE_RCU.
Would that look clearer?
Thanks.
Mark Rutland
reviewerRe: [PATCH 10/21] context_tracking: Take idle eqs entrypoints over RCU
Quoted 16 lines · Frederic Weisbecker
Quoted 8 lines · Mark Rutland
Hi Frederic,
Quoted 4 lines · Frederic Weisbecker
The RCU dynticks counter is going to be merged into the context tracking subsystem. Start with moving the idle extended quiescent states entrypoints to context tracking. For now those are dumb redirections to existing RCU calls.
Quoted 6 lines
I was a bit confused looking at this, because that redirection only exists for
CONFIG_CONTEXT_TRACKING, and is empty otherwise.I see this patch makes
TREE_RCUselectCONTEXT_TRACKING, which means that works. Since that also means building the rest of the context tracking code, I think it'd be worth mentioning that in the commit message.
Quoted 5 lines
Not exactly, it only builds a minimal version of
CONTEXT_TRACKING, which maps to roughly the vanilla RCU-idle code.The user tracking part (which is the vanilla
CONTEXT_TRACKING) now becomesCONFIG_CONTEXT_TRACKING_USER
Ah, sorry. That makes sense to me now!
Quoted 10 lines
Quoted 2 lines
Do all architectures which can use
TREE_RCUtoday already support context tracking? If not, do those work by default?No need to, the old context tracking becomes
CONFIG_CONTEXT_TRACKING_USERwhich is not needed byTREE_RCU.Ok you're right, this is all very confusing :o)
Let's clarify this, I'm going to do the following on the next version:
Quoted 8 lines
* The current
CONFIG_CONTEXT_TRACKINGbecomesCONFIG_CONTEXT_TRACKING_USER. This part needs support from the architecture.* The RCU-idle functions that have been moved to context tracking will become
CONFIG_CONTEXT_TRACKING_IDLE. This part doesn't need support from the architecture and will be set byTREE_RCU.Would that look clearer?
I think so, yes!
It would also help if the prior commit message could be explicit about the end result: After the Kconfig rework both CONFIG_CONTEXT_TRACKING_USER and CONFIG_CONTEXT_TRACKING exist, but CONFIG_CONTEXT_TRACKING will only used to decide whether to build object files with common context tracking logic.
Thanks,
Mark.
Frederic Weisbecker
maintainerPATCH 11/21 context_tracking: Take IRQ eqs entrypoints over RCU
In reply to Frederic WeisbeckerThe RCU dynticks counter is going to be merged into the context tracking subsystem. Prepare with moving the IRQ extended quiescent states entrypoints to context tracking. For now those are dumb redirection to existing RCU calls.
--- .../RCU/Design/Requirements/Requirements.rst | 10 ++++---- Documentation/RCU/stallwarn.rst | 4 ++-- arch/Kconfig | 2 +- arch/arm64/kernel/entry-common.c | 6 ++--- arch/x86/mm/fault.c | 2 +- drivers/cpuidle/cpuidle-psci.c | 8 +++---- drivers/cpuidle/cpuidle-riscv-sbi.c | 8 +++---- include/linux/context_tracking_irq.h | 17 +++++++++++++ include/linux/context_tracking_state.h | 1 + include/linux/entry-common.h | 10 ++++---- include/linux/rcupdate.h | 5 ++-- include/linux/tracepoint.h | 4 ++-- kernel/context_tracking.c | 24 +++++++++++++++++-- kernel/cpu_pm.c | 8 +++---- kernel/entry/common.c | 12 +++++----- kernel/softirq.c | 4 ++-- kernel/trace/trace.c | 6 ++--- 17 files changed, 85 insertions(+), 46 deletions(-) create mode 100644 include/linux/context_tracking_irq.h
Documentation/RCU/Design/Requirements/Requirements.rst +5 -5
@@ -1844,10 +1844,10 @@ that meets this requirement. Furthermore, NMI handlers can be interrupted by what appear to RCU to be normal interrupts. One way that this can happen is for code that -directly invokes rcu_irq_enter() and rcu_irq_exit() to be called +directly invokes ct_irq_enter() and ct_irq_exit() to be called from an NMI handler. This astonishing fact of life prompted the current -code structure, which has rcu_irq_enter() invoking -rcu_nmi_enter() and rcu_irq_exit() invoking rcu_nmi_exit(). +code structure, which has ct_irq_enter() invoking +rcu_nmi_enter() and ct_irq_exit() invoking rcu_nmi_exit(). And yes, I also learned of this requirement the hard way. Loadable Modules
@@ -2195,7 +2195,7 @@ scheduling-clock interrupt be enabled when RCU needs it to be: sections, and RCU believes this CPU to be idle, no problem. This sort of thing is used by some architectures for light-weight exception handlers, which can then avoid the overhead of - rcu_irq_enter() and rcu_irq_exit() at exception entry and + ct_irq_enter() and ct_irq_exit() at exception entry and exit, respectively. Some go further and avoid the entireties of irq_enter() and irq_exit(). Just make very sure you are running some of your tests with
@@ -2226,7 +2226,7 @@ scheduling-clock interrupt be enabled when RCU needs it to be: +-----------------------------------------------------------------------+ | **Answer**: | +-----------------------------------------------------------------------+ -| One approach is to do ``rcu_irq_exit();rcu_irq_enter();`` every so | +| One approach is to do ``ct_irq_exit();ct_irq_enter();`` every so | | often. But given that long-running interrupt handlers can cause other | | problems, not least for response time, shouldn't you work to keep | | your interrupt handler's runtime within reasonable bounds? |
Documentation/RCU/stallwarn.rst +2 -2
@@ -98,11 +98,11 @@ warnings: - A low-level kernel issue that either fails to invoke one of the variants of rcu_user_enter(), rcu_user_exit(), ct_idle_enter(), - ct_idle_exit(), rcu_irq_enter(), or rcu_irq_exit() on the one + ct_idle_exit(), ct_irq_enter(), or ct_irq_exit() on the one hand, or that invokes one of them too many times on the other. Historically, the most frequent issue has been an omission of either irq_enter() or irq_exit(), which in turn invoke - rcu_irq_enter() or rcu_irq_exit(), respectively. Building your + ct_irq_enter() or ct_irq_exit(), respectively. Building your kernel with CONFIG_RCU_EQS_DEBUG=y can help track down these types of issues, which sometimes arise in architecture-specific code.
arch/Kconfig +1 -1
@@ -777,7 +777,7 @@ config HAVE_CONTEXT_TRACKING_USER Syscalls need to be wrapped inside user_exit()-user_enter(), either optimized behind static key or through the slow path using TIF_NOHZ flag. Exceptions handlers must be wrapped as well. Irqs are already - protected inside rcu_irq_enter/rcu_irq_exit() but preemption or signal + protected inside ct_irq_enter/ct_irq_exit() but preemption or signal handling on irq exit still need to be protected. config HAVE_CONTEXT_TRACKING_USER_OFFSTACK
arch/arm64/kernel/entry-common.c +3 -3
@@ -41,7 +41,7 @@ static __always_inline void __enter_from_kernel_mode(struct pt_regs *regs)if (!IS_ENABLED (CONFIG_TINY_RCU ) &&is_idle_task (current)) {lockdep_hardirqs_off (CALLER_ADDR0 ); -rcu_irq_enter (); +ct_irq_enter ();trace_hardirqs_off_finish (); regs->exit_rcu = true;
@@ -76,7 +76,7 @@ static __always_inline void __exit_to_kernel_mode(struct pt_regs *regs)if (regs->exit_rcu) {trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); -rcu_irq_exit (); +ct_irq_exit ();lockdep_hardirqs_on (CALLER_ADDR0 );return ; }
@@ -84,7 +84,7 @@ static __always_inline void __exit_to_kernel_mode(struct pt_regs *regs)trace_hardirqs_on (); }else {if (regs->exit_rcu) -rcu_irq_exit (); +ct_irq_exit (); } }
arch/x86/mm/fault.c +1 -1
@@ -1526,7 +1526,7 @@ DEFINE_IDTENTRY_RAW_ERRORCODE(exc_page_fault)/* * Entry handling for valid #PF from kernel mode is slightly -* different: RCU is already watching and rcu_irq_enter() must not +* different: RCU is already watching and ct_irq_enter() must not * be invoked because a kernel fault on a user space address might * sleep. *
drivers/cpuidle/cpuidle-psci.c +4 -4
@@ -68,12 +68,12 @@ static int __psci_enter_domain_idle_state(struct cpuidle_device *dev,return -1 ;/* Do runtime PM to manage a hierarchical CPU toplogy. */ -rcu_irq_enter_irqson (); +ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_suspend (pd_dev);else pm_runtime_put_sync_suspend (pd_dev); -rcu_irq_exit_irqson (); +ct_irq_exit_irqson (); state =psci_get_domain_state ();if (!state)
@@ -81,12 +81,12 @@ static int __psci_enter_domain_idle_state(struct cpuidle_device *dev, ret =psci_cpu_suspend_enter (state) ?-1 : idx; -rcu_irq_enter_irqson (); +ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_resume (pd_dev);else pm_runtime_get_sync (pd_dev); -rcu_irq_exit_irqson (); +ct_irq_exit_irqson ();cpu_pm_exit ();
drivers/cpuidle/cpuidle-riscv-sbi.c +4 -4
@@ -115,12 +115,12 @@ static int __sbi_enter_domain_idle_state(struct cpuidle_device *dev,return -1 ;/* Do runtime PM to manage a hierarchical CPU toplogy. */ -rcu_irq_enter_irqson (); +ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_suspend (pd_dev);else pm_runtime_put_sync_suspend (pd_dev); -rcu_irq_exit_irqson (); +ct_irq_exit_irqson ();if (sbi_is_domain_state_available ()) state =sbi_get_domain_state ();
@@ -129,12 +129,12 @@ static int __sbi_enter_domain_idle_state(struct cpuidle_device *dev, ret =sbi_suspend (state) ?-1 : idx; -rcu_irq_enter_irqson (); +ct_irq_enter_irqson ();if (s2idle)dev_pm_genpd_resume (pd_dev);else pm_runtime_get_sync (pd_dev); -rcu_irq_exit_irqson (); +ct_irq_exit_irqson ();cpu_pm_exit ();
include/linux/context_tracking_irq.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_CONTEXT_TRACKING_IRQ_H +#define _LINUX_CONTEXT_TRACKING_IRQ_H + +#ifdef CONFIG_CONTEXT_TRACKING +void ct_irq_enter (void ); +void ct_irq_exit (void ); +void ct_irq_enter_irqson (void ); +void ct_irq_exit_irqson (void ); +#else +static inline void ct_irq_enter (void ) { } +static inline void ct_irq_exit (void ) { } +static inline void ct_irq_enter_irqson (void ) { } +static inline void ct_irq_exit_irqson (void ) { } +#endif + +#endif
include/linux/context_tracking_state.h +1 -0
@@ -4,6 +4,7 @@#include <linux/percpu.h> #include <linux/static_key.h> +#include <linux/context_tracking_irq.h> struct context_tracking {/*
include/linux/entry-common.h +5 -5
@@ -357,7 +357,7 @@ void irqentry_exit_to_user_mode(struct pt_regs *regs);/** * struct irqentry_state - Opaque object for exception state storage * @exit_rcu: Used exclusively in the irqentry_*() calls; signals whether the -* exit path has to invoke rcu_irq_exit(). +* exit path has to invoke ct_irq_exit(). * @lockdep: Used exclusively in the irqentry_nmi_*() calls; ensures that * lockdep state is restored correctly on exit from nmi. *
@@ -395,12 +395,12 @@ typedef struct irqentry_state {* * For kernel mode entries RCU handling is done conditional. If RCU is * watching then the only RCU requirement is to check whether the tick has -* to be restarted. If RCU is not watching then rcu_irq_enter() has to be -* invoked on entry and rcu_irq_exit() on exit. +* to be restarted. If RCU is not watching then ct_irq_enter() has to be +* invoked on entry and ct_irq_exit() on exit. * -* Avoiding the rcu_irq_enter/exit() calls is an optimization but also +* Avoiding the ct_irq_enter/exit() calls is an optimization but also * solves the problem of kernel mode pagefaults which can schedule, which -* is not possible after invoking rcu_irq_enter() without undoing it. +* is not possible after invoking ct_irq_enter() without undoing it. * * For user mode entries irqentry_enter_from_user_mode() is invoked to * establish the proper context for NOHZ_FULL. Otherwise scheduling on exit
include/linux/rcupdate.h +3 -2
@@ -29,6 +29,7 @@#include <linux/lockdep.h> #include <asm/processor.h> #include <linux/cpumask.h> +#include <linux/context_tracking_irq.h> #define ULONG_CMP_GE (a, b) (ULONG_MAX / 2 >= (a) - (b))#define ULONG_CMP_LT (a, b) (ULONG_MAX / 2 < (a) - (b))
@@ -144,9 +145,9 @@ static inline void rcu_nocb_flush_deferred_wakeup(void) { }*/ #define RCU_NONIDLE (a) \ do { \ - rcu_irq_enter_irqson(); \ + ct_irq_enter_irqson(); \ do { a; } while (0); \ - rcu_irq_exit_irqson(); \ + ct_irq_exit_irqson(); \ } while (0)/*
include/linux/tracepoint.h +2 -2
@@ -200,13 +200,13 @@ static inline struct tracepoint *tracepoint_ptr_deref(tracepoint_ptr_t *p)*/ \if (rcuidle) { \ __idx =srcu_read_lock_notrace (&tracepoint_srcu);\ -rcu_irq_enter_irqson (); \ +ct_irq_enter_irqson (); \ } \ \__DO_TRACE_CALL (name,TP_ARGS (args)); \ \if (rcuidle) { \ -rcu_irq_exit_irqson (); \ +ct_irq_exit_irqson (); \srcu_read_unlock_notrace (&tracepoint_srcu, __idx);\ } \ \
kernel/context_tracking.c +22 -2
@@ -75,7 +75,7 @@ void noinstr __ct_user_enter(enum ctx_state state)* At this stage, only low level arch entry code remains and * then we'll run in userspace. We can assume there won't be * any RCU read-side critical section until the next call to -* user_exit() or rcu_irq_enter(). Let's remove RCU's dependency +* user_exit() or ct_irq_enter(). Let's remove RCU's dependency * on the tick. */ if (state ==CONTEXT_USER ) {
@@ -117,7 +117,7 @@ void ct_user_enter(enum ctx_state state)/* * Some contexts may involve an exception occuring in an irq, * leading to that nesting: -* rcu_irq_enter() rcu_user_exit() rcu_user_exit() rcu_irq_exit() +* ct_irq_enter() rcu_user_exit() rcu_user_exit() ct_irq_exit() * This would mess up the dyntick_nesting count though. And rcu_irq_*() * helpers are enough to protect RCU uses inside the exception. So * just return immediately if we detect we are in an IRQ.
@@ -263,3 +263,23 @@ void ct_idle_exit(void)rcu_idle_exit (); }EXPORT_SYMBOL_GPL (ct_idle_exit); + +noinstr void ct_irq_enter(void) +{ +rcu_irq_enter (); +} + +noinstr void ct_irq_exit(void) +{ +rcu_irq_exit (); +} + +void ct_irq_enter_irqson (void ) +{ +rcu_irq_enter_irqson (); +} + +void ct_irq_exit_irqson (void ) +{ +rcu_irq_exit_irqson (); +}
kernel/cpu_pm.c +4 -4
@@ -35,11 +35,11 @@ static int cpu_pm_notify(enum cpu_pm_event event)* disfunctional in cpu idle. Copy RCU_NONIDLE code to let RCU know * this. */ -rcu_irq_enter_irqson (); +ct_irq_enter_irqson ();rcu_read_lock (); ret =raw_notifier_call_chain (&cpu_pm_notifier.chain, event,NULL );rcu_read_unlock (); -rcu_irq_exit_irqson (); +ct_irq_exit_irqson ();return notifier_to_errno (ret); }
@@ -49,11 +49,11 @@ static int cpu_pm_notify_robust(enum cpu_pm_event event_up, enum cpu_pm_event evunsigned long flags;int ret; -rcu_irq_enter_irqson (); +ct_irq_enter_irqson ();raw_spin_lock_irqsave (&cpu_pm_notifier.lock, flags); ret =raw_notifier_call_chain_robust (&cpu_pm_notifier.chain, event_up, event_down,NULL );raw_spin_unlock_irqrestore (&cpu_pm_notifier.lock, flags); -rcu_irq_exit_irqson (); +ct_irq_exit_irqson ();return notifier_to_errno (ret); }
kernel/entry/common.c +6 -6
@@ -321,7 +321,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs) }/* -* If this entry hit the idle task invoke rcu_irq_enter() whether +* If this entry hit the idle task invoke ct_irq_enter() whether * RCU is watching or not. * * Interrupts can nest when the first interrupt invokes softirq
@@ -332,12 +332,12 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)* not nested into another interrupt. * * Checking for rcu_is_watching() here would prevent the nesting -* interrupt to invoke rcu_irq_enter(). If that nested interrupt is +* interrupt to invoke ct_irq_enter(). If that nested interrupt is * the tick then rcu_flavor_sched_clock_irq() would wrongfully * assume that it is the first interrupt and eventually claim * quiescent state and end grace periods prematurely. * -* Unconditionally invoke rcu_irq_enter() so RCU state stays +* Unconditionally invoke ct_irq_enter() so RCU state stays * consistent. * * TINY_RCU does not support EQS, so let the compiler eliminate
@@ -350,7 +350,7 @@ noinstr irqentry_state_t irqentry_enter(struct pt_regs *regs)* as in irqentry_enter_from_user_mode(). */ lockdep_hardirqs_off (CALLER_ADDR0 ); -rcu_irq_enter (); +ct_irq_enter ();instrumentation_begin ();trace_hardirqs_off_finish ();instrumentation_end ();
@@ -418,7 +418,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)trace_hardirqs_on_prepare ();lockdep_hardirqs_on_prepare (CALLER_ADDR0 );instrumentation_end (); -rcu_irq_exit (); +ct_irq_exit ();lockdep_hardirqs_on (CALLER_ADDR0 );return ; }
@@ -436,7 +436,7 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state)* was not watching on entry. */ if (state.exit_rcu) -rcu_irq_exit (); +ct_irq_exit (); } }
kernel/softirq.c +2 -2
@@ -607,7 +607,7 @@ void irq_enter_rcu(void)*/ void irq_enter (void ) { -rcu_irq_enter (); +ct_irq_enter ();irq_enter_rcu (); }
@@ -659,7 +659,7 @@ void irq_exit_rcu(void)void irq_exit (void ) {__irq_exit_rcu (); -rcu_irq_exit (); +ct_irq_exit ();/* must be last! */ lockdep_hardirq_exit (); }
kernel/trace/trace.c +3 -3
@@ -3104,15 +3104,15 @@ void __trace_stack(struct trace_array *tr, unsigned int trace_ctx,/* * When an NMI triggers, RCU is enabled via rcu_nmi_enter(), * but if the above rcu_is_watching() failed, then the NMI -* triggered someplace critical, and rcu_irq_enter() should +* triggered someplace critical, and ct_irq_enter() should * not be called from NMI. */ if (unlikely (in_nmi ()))return ; -rcu_irq_enter_irqson (); +ct_irq_enter_irqson ();__ftrace_trace_stack (buffer, trace_ctx, skip,NULL ); -rcu_irq_exit_irqson (); +ct_irq_exit_irqson (); }/**
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 12/21 context_tracking: Take NMI eqs entrypoints over RCU
In reply to Frederic WeisbeckerThe RCU dynticks counter is going to be merged into the context tracking subsystem. Prepare with moving the NMI extended quiescent states entrypoints to context tracking. For now those are dumb redirection to existing RCU calls.
--- Documentation/RCU/Design/Requirements/Requirements.rst | 2 +- arch/Kconfig | 2 +- arch/arm64/kernel/entry-common.c | 8 ++++---- include/linux/context_tracking_irq.h | 4 ++++ include/linux/hardirq.h | 4 ++-- kernel/context_tracking.c | 10 ++++++++++ kernel/entry/common.c | 4 ++-- kernel/extable.c | 4 ++-- kernel/trace/trace.c | 2 +- 9 files changed, 27 insertions(+), 13 deletions(-)
Documentation/RCU/Design/Requirements/Requirements.rst +1 -1
@@ -1847,7 +1847,7 @@ normal interrupts. One way that this can happen is for code that directly invokes ct_irq_enter() and ct_irq_exit() to be called from an NMI handler. This astonishing fact of life prompted the current code structure, which has ct_irq_enter() invoking -rcu_nmi_enter() and ct_irq_exit() invoking rcu_nmi_exit(). +ct_nmi_enter() and ct_irq_exit() invoking ct_nmi_exit(). And yes, I also learned of this requirement the hard way. Loadable Modules
arch/Kconfig +1 -1
@@ -792,7 +792,7 @@ config HAVE_CONTEXT_TRACKING_USER_OFFSTACK - Critical entry code isn't preemptible (or better yet: not interruptible). - - No use of RCU read side critical sections, unless rcu_nmi_enter() + - No use of RCU read side critical sections, unless ct_nmi_enter() got called. - No use of instrumentation, unless instrumentation_begin() got called.
arch/arm64/kernel/entry-common.c +4 -4
@@ -161,7 +161,7 @@ static void noinstr arm64_enter_nmi(struct pt_regs *regs)__nmi_enter ();lockdep_hardirqs_off (CALLER_ADDR0 );lockdep_hardirq_enter (); -rcu_nmi_enter (); +ct_nmi_enter ();trace_hardirqs_off_finish ();ftrace_nmi_enter ();
@@ -182,7 +182,7 @@ static void noinstr arm64_exit_nmi(struct pt_regs *regs)lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); } -rcu_nmi_exit (); +ct_nmi_exit ();lockdep_hardirq_exit ();if (restore)lockdep_hardirqs_on (CALLER_ADDR0 );
@@ -199,7 +199,7 @@ static void noinstr arm64_enter_el1_dbg(struct pt_regs *regs) regs->lockdep_hardirqs =lockdep_hardirqs_enabled ();lockdep_hardirqs_off (CALLER_ADDR0 ); -rcu_nmi_enter (); +ct_nmi_enter ();trace_hardirqs_off_finish (); }
@@ -218,7 +218,7 @@ static void noinstr arm64_exit_el1_dbg(struct pt_regs *regs)lockdep_hardirqs_on_prepare (CALLER_ADDR0 ); } -rcu_nmi_exit (); +ct_nmi_exit ();if (restore)lockdep_hardirqs_on (CALLER_ADDR0 ); }
include/linux/context_tracking_irq.h +4 -0
@@ -7,11 +7,15 @@ void ct_irq_enter(void);void ct_irq_exit (void );void ct_irq_enter_irqson (void );void ct_irq_exit_irqson (void ); +void ct_nmi_enter (void ); +void ct_nmi_exit (void );#else static inline void ct_irq_enter (void ) { }static inline void ct_irq_exit (void ) { }static inline void ct_irq_enter_irqson (void ) { }static inline void ct_irq_exit_irqson (void ) { } +static inline void ct_nmi_enter (void ) { } +static inline void ct_nmi_exit (void ) { }#endif #endif
include/linux/hardirq.h +2 -2
@@ -124,7 +124,7 @@ extern void rcu_nmi_exit(void); do { \ __nmi_enter(); \ lockdep_hardirq_enter(); \ - rcu_nmi_enter(); \ + ct_nmi_enter(); \ instrumentation_begin(); \ ftrace_nmi_enter(); \ instrumentation_end(); \
@@ -143,7 +143,7 @@ extern void rcu_nmi_exit(void);instrumentation_begin (); \ftrace_nmi_exit (); \instrumentation_end (); \ -rcu_nmi_exit (); \ +ct_nmi_exit (); \lockdep_hardirq_exit (); \__nmi_exit (); \ }while (0 )
kernel/context_tracking.c +10 -0
@@ -283,3 +283,13 @@ void ct_irq_exit_irqson(void) {rcu_irq_exit_irqson (); } + +noinstr void ct_nmi_enter(void) +{ +rcu_nmi_enter (); +} + +noinstr void ct_nmi_exit(void) +{ +rcu_nmi_exit (); +}
kernel/entry/common.c +2 -2
@@ -449,7 +449,7 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs)__nmi_enter ();lockdep_hardirqs_off (CALLER_ADDR0 );lockdep_hardirq_enter (); -rcu_nmi_enter (); +ct_nmi_enter ();instrumentation_begin ();trace_hardirqs_off_finish ();
@@ -469,7 +469,7 @@ void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state) }instrumentation_end (); -rcu_nmi_exit (); +ct_nmi_exit ();lockdep_hardirq_exit ();if (irq_state.lockdep)lockdep_hardirqs_on (CALLER_ADDR0 );
kernel/extable.c +2 -2
@@ -114,7 +114,7 @@ int kernel_text_address(unsigned long addr)/* Treat this like an NMI as it can happen anywhere */ if (no_rcu) -rcu_nmi_enter (); +ct_nmi_enter ();if (is_module_text_address (addr)) goto out;
@@ -127,7 +127,7 @@ int kernel_text_address(unsigned long addr) ret =0 ; out:if (no_rcu) -rcu_nmi_exit (); +ct_nmi_exit ();return ret; }
kernel/trace/trace.c +1 -1
@@ -3102,7 +3102,7 @@ void __trace_stack(struct trace_array *tr, unsigned int trace_ctx, }/* -* When an NMI triggers, RCU is enabled via rcu_nmi_enter(), +* When an NMI triggers, RCU is enabled via ct_nmi_enter(), * but if the above rcu_is_watching() failed, then the NMI * triggered someplace critical, and ct_irq_enter() should * not be called from NMI.
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 13/21 rcu/context-tracking: Remove rcu_irq_enter/exit()
In reply to Frederic WeisbeckerNow rcu_irq_enter/exit() is an unecessary middle call between ct_irq_enter/exit() and nmi_irq_enter/exit(). Take this opportunity to remove the former functions and move the comments above them to the new entrypoints.
--- include/linux/rcutiny.h | 4 -- include/linux/rcutree.h | 4 -- kernel/context_tracking.c | 79 +++++++++++++++++++++++++++++++-- kernel/rcu/tree.c | 91 --------------------------------------- 4 files changed, 75 insertions(+), 103 deletions(-)
include/linux/rcutiny.h +0 -4
@@ -88,10 +88,6 @@ static inline void rcu_cpu_stall_reset(void) { }static inline int rcu_jiffies_till_stall_check (void ) {return 21 *HZ ; }static inline void rcu_idle_enter (void ) { }static inline void rcu_idle_exit (void ) { } -static inline void rcu_irq_enter (void ) { } -static inline void rcu_irq_exit_irqson (void ) { } -static inline void rcu_irq_enter_irqson (void ) { } -static inline void rcu_irq_exit (void ) { }static inline void rcu_irq_exit_check_preempt (void ) { }#define rcu_is_idle_cpu (cpu) \ (is_idle_task(current) && !in_nmi() && !in_hardirq() && !in_serving_softirq())
include/linux/rcutree.h +0 -4
@@ -49,10 +49,6 @@ void cond_synchronize_rcu(unsigned long oldstate);void rcu_idle_enter (void );void rcu_idle_exit (void ); -void rcu_irq_enter (void ); -void rcu_irq_exit (void ); -void rcu_irq_enter_irqson (void ); -void rcu_irq_exit_irqson (void );bool rcu_is_idle_cpu (int cpu);#ifdef CONFIG_PROVE_RCU
kernel/context_tracking.c +75 -4
99 lines not on this page · raw message
kernel/rcu/tree.c +0 -91
109 lines not on this page · raw message
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 14/21 rcu/context_tracking: Move dynticks counter to context tracking
In reply to Frederic WeisbeckerIn order to prepare for merging RCU dynticks counter into the context tracking state, move the rcu_data's dynticks field to the context tracking structure. It will later be mixed within the context tracking state itself.
--- include/linux/context_tracking_state.h | 27 ++++++++++++- kernel/context_tracking.c | 9 +++-- kernel/rcu/tree.c | 56 +++++++++++++------------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_stall.h | 4 +- 6 files changed, 61 insertions(+), 38 deletions(-)
include/linux/context_tracking_state.h +25 -2
@@ -7,6 +7,7 @@#include <linux/context_tracking_irq.h> struct context_tracking { +#ifdef CONFIG_CONTEXT_TRACKING_USER /* * When active is false, probes are unset in order * to minimize overhead: TIF flags are cleared
@@ -21,12 +22,34 @@ struct context_tracking {CONTEXT_USER ,CONTEXT_GUEST , } state; +#endif +atomic_t dynticks;/* Even value for idle, else odd. */ }; -#ifdef CONFIG_CONTEXT_TRACKING_USER -extern struct static_key_false context_tracking_key; +#ifdef CONFIG_CONTEXT_TRACKING DECLARE_PER_CPU(struct context_tracking, context_tracking); +static __always_inline int ct_dynticks(void) +{ +return atomic_read (this_cpu_ptr (&context_tracking.dynticks)); +} + +static __always_inline int ct_dynticks_cpu(int cpu) +{ +struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu); +return atomic_read (&ct->dynticks); +} + +static __always_inline int ct_dynticks_cpu_acquire(int cpu) +{ +struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu); +return atomic_read_acquire (&ct->state); +} +#endif + +#ifdef CONFIG_CONTEXT_TRACKING_USER +extern struct static_key_false context_tracking_key; + static __always_inline bool context_tracking_enabled(void) {return static_branch_unlikely (&context_tracking_key);
kernel/context_tracking.c +6 -3
@@ -30,9 +30,6 @@DEFINE_STATIC_KEY_FALSE (context_tracking_key);EXPORT_SYMBOL_GPL (context_tracking_key); -DEFINE_PER_CPU(struct context_tracking, context_tracking); -EXPORT_SYMBOL_GPL (context_tracking); - static noinstr bool context_tracking_recursion_enter(void) {int recursion;
@@ -252,6 +249,12 @@ void __init context_tracking_init(void)#endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ +DEFINE_PER_CPU(struct context_tracking, context_tracking) = { + .dynticks =ATOMIC_INIT (1 ), +}; +EXPORT_SYMBOL_GPL (context_tracking); + + noinstr void ct_idle_enter(void) {rcu_idle_enter ();
kernel/rcu/tree.c +27 -29
208 lines not on this page · raw message
kernel/rcu/tree.h +0 -1
@@ -193,7 +193,6 @@ struct rcu_data {int dynticks_snap;/* Per-GP tracking for dynticks. */ long dynticks_nesting;/* Track process nesting level. */ long dynticks_nmi_nesting;/* Track irq/NMI nesting level. */ -atomic_t dynticks;/* Even value for idle, else odd. */ bool rcu_need_heavy_qs;/* GP old, so heavy quiescent state! */ bool rcu_urgent_qs;/* GP old need light quiescent state. */ bool rcu_forced_tick;/* Forced tick to provide QS. */
kernel/rcu/tree_exp.h +1 -1
@@ -358,7 +358,7 @@ static void __sync_rcu_exp_select_node_cpus(struct rcu_exp_work *rewp) !(rnp->qsmaskinitnext & mask)) { mask_ofl_test |= mask; } else { - snap =rcu_dynticks_snap (rdp); + snap =rcu_dynticks_snap (cpu);if (rcu_dynticks_in_eqs (snap)) mask_ofl_test |= mask;else
kernel/rcu/tree_stall.h +2 -2
@@ -463,7 +463,7 @@ static void print_cpu_stall_info(int cpu) } delta =rcu_seq_ctr (rdp->mynode->gp_seq - rdp->rcu_iw_gp_seq); falsepositive =rcu_is_gp_kthread_starving (NULL ) && -rcu_dynticks_in_eqs (rcu_dynticks_snap (rdp)); +rcu_dynticks_in_eqs (rcu_dynticks_snap (cpu)); rcuc_starved =rcu_is_rcuc_kthread_starving (rdp, &j);if (rcuc_starved)sprintf (buf," rcuc=%ld jiffies(starved)" , j);
@@ -476,7 +476,7 @@ static void print_cpu_stall_info(int cpu) rdp->rcu_iw_pending ? (int )min (delta,9UL ) +'0' :"!." [!delta], ticks_value, ticks_title, -rcu_dynticks_snap (rdp) &0xfff , +rcu_dynticks_snap (cpu) &0xfff , rdp->dynticks_nesting, rdp->dynticks_nmi_nesting, rdp->softirq_snap,kstat_softirqs_cpu (RCU_SOFTIRQ , cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart,
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 15/21 rcu/context_tracking: Move dynticks_nesting to context tracking
In reply to Frederic WeisbeckerThe RCU eqs tracking is going to be performed by the context tracking subsystem. The related nesting counters thus need to be moved to the context tracking structure.
--- include/linux/context_tracking_state.h | 12 ++++++++++ kernel/context_tracking.c | 1 + kernel/rcu/tree.c | 31 +++++++++++++------------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_stall.h | 2 +- 5 files changed, 30 insertions(+), 17 deletions(-)
include/linux/context_tracking_state.h +12 -0
@@ -24,6 +24,7 @@ struct context_tracking { } state;#endif atomic_t dynticks;/* Even value for idle, else odd. */ +long dynticks_nesting;/* Track process nesting level. */ };#ifdef CONFIG_CONTEXT_TRACKING
@@ -45,6 +46,17 @@ static __always_inline int ct_dynticks_cpu_acquire(int cpu)struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu); return atomic_read_acquire(&ct->state); } + +static __always_inline long ct_dynticks_nesting (void ) +{ +return __this_cpu_read (context_tracking.dynticks_nesting); +} + +static __always_inline long ct_dynticks_nesting_cpu (int cpu) +{ +struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu); +return ct->dynticks_nesting; +}#endif #ifdef CONFIG_CONTEXT_TRACKING_USER
kernel/context_tracking.c +1 -0
@@ -250,6 +250,7 @@ void __init context_tracking_init(void)#endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ DEFINE_PER_CPU(struct context_tracking, context_tracking) = { + .dynticks_nesting =1 , .dynticks =ATOMIC_INIT (1 ), };EXPORT_SYMBOL_GPL (context_tracking);
kernel/rcu/tree.c +16 -15
@@ -75,7 +75,6 @@/* Data structures. */ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = { - .dynticks_nesting =1 , .dynticks_nmi_nesting =DYNTICK_IRQ_NONIDLE , #ifdef CONFIG_RCU_NOCB_CPU .cblist.flags = SEGCBLIST_RCU_CORE,
@@ -436,7 +435,7 @@ static int rcu_is_cpu_rrupt_from_idle(void)lockdep_assert_irqs_disabled ();/* Check for counter underflows */ -RCU_LOCKDEP_WARN (__this_cpu_read (rcu_data.dynticks_nesting) <0 , +RCU_LOCKDEP_WARN (ct_dynticks_nesting () <0 ,"RCU dynticks_nesting counter underflow!" );RCU_LOCKDEP_WARN (__this_cpu_read (rcu_data.dynticks_nmi_nesting) <=0 ,"RCU dynticks_nmi_nesting counter underflow/zero!" );
@@ -452,7 +451,7 @@ static int rcu_is_cpu_rrupt_from_idle(void)WARN_ON_ONCE (!nesting && !is_idle_task (current));/* Does CPU appear to be idle from an RCU standpoint? */ -return __this_cpu_read (rcu_data.dynticks_nesting) ==0 ; +return ct_dynticks_nesting () ==0 ; }#define DEFAULT_RCU_BLIMIT (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) ? 1000 : 10)
@@ -619,16 +618,16 @@ static noinstr void rcu_eqs_enter(bool user)WARN_ON_ONCE (rdp->dynticks_nmi_nesting !=DYNTICK_IRQ_NONIDLE );WRITE_ONCE (rdp->dynticks_nmi_nesting,0 );WARN_ON_ONCE (IS_ENABLED (CONFIG_RCU_EQS_DEBUG ) && - rdp->dynticks_nesting ==0 ); -if (rdp->dynticks_nesting !=1 ) { +ct_dynticks_nesting () ==0 ); +if (ct_dynticks_nesting () !=1 ) {// RCU will still be watching, so just do accounting and leave. - rdp->dynticks_nesting--; + ct->dynticks_nesting--;return ; }lockdep_assert_irqs_disabled ();instrumentation_begin (); -trace_rcu_dyntick (TPS ("Start" ), rdp->dynticks_nesting,0 ,ct_dynticks ()); +trace_rcu_dyntick (TPS ("Start" ),ct_dynticks_nesting (),0 ,ct_dynticks ());WARN_ON_ONCE (IS_ENABLED (CONFIG_RCU_EQS_DEBUG ) && !user && !is_idle_task (current));rcu_preempt_deferred_qs (current);
@@ -636,7 +635,7 @@ static noinstr void rcu_eqs_enter(bool user)instrument_atomic_write (&ct->dynticks,sizeof (ct->dynticks));instrumentation_end (); -WRITE_ONCE (rdp->dynticks_nesting,0 );/* Avoid irq-access tearing. */ +WRITE_ONCE (ct->dynticks_nesting,0 );/* Avoid irq-access tearing. */ // RCU is watching here ... rcu_dynticks_eqs_enter ();// ... but is no longer watching here.
@@ -793,7 +792,7 @@ void rcu_irq_exit_check_preempt(void) {lockdep_assert_irqs_disabled (); -RCU_LOCKDEP_WARN (__this_cpu_read (rcu_data.dynticks_nesting) <=0 , +RCU_LOCKDEP_WARN (ct_dynticks_nesting () <=0 ,"RCU dynticks_nesting counter underflow/zero!" ); RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) != DYNTICK_IRQ_NONIDLE,
@@ -819,11 +818,11 @@ static void noinstr rcu_eqs_exit(bool user)lockdep_assert_irqs_disabled (); rdp =this_cpu_ptr (&rcu_data); - oldval = rdp->dynticks_nesting; + oldval =ct_dynticks_nesting ();WARN_ON_ONCE (IS_ENABLED (CONFIG_RCU_EQS_DEBUG ) && oldval <0 );if (oldval) {// RCU was already watching, so just do accounting and leave. - rdp->dynticks_nesting++; + ct->dynticks_nesting++;return ; }rcu_dynticks_task_exit ();
@@ -835,9 +834,9 @@ static void noinstr rcu_eqs_exit(bool user)// instrumentation for the noinstr rcu_dynticks_eqs_exit() instrument_atomic_write (&ct->dynticks,sizeof (ct->dynticks)); -trace_rcu_dyntick (TPS ("End" ), rdp->dynticks_nesting,1 ,ct_dynticks ()); +trace_rcu_dyntick (TPS ("End" ),ct_dynticks_nesting (),1 ,ct_dynticks ());WARN_ON_ONCE (IS_ENABLED (CONFIG_RCU_EQS_DEBUG ) && !user && !is_idle_task (current)); -WRITE_ONCE (rdp->dynticks_nesting,1 ); +WRITE_ONCE (ct->dynticks_nesting,1 );WARN_ON_ONCE (rdp->dynticks_nmi_nesting);WRITE_ONCE (rdp->dynticks_nmi_nesting,DYNTICK_IRQ_NONIDLE );instrumentation_end ();
@@ -4229,12 +4228,13 @@ static void rcu_init_new_rnp(struct rcu_node *rnp_leaf)static void __init rcu_boot_init_percpu_data (int cpu) { +struct context_tracking *ct =this_cpu_ptr (&context_tracking);struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu);/* Set up local state, ensuring consistent view of global state. */ rdp->grpmask =leaf_node_cpu_bit (rdp->mynode, cpu);INIT_WORK (&rdp->strict_work, strict_work_handler); -WARN_ON_ONCE (rdp->dynticks_nesting !=1 ); +WARN_ON_ONCE (ct->dynticks_nesting !=1 );WARN_ON_ONCE (rcu_dynticks_in_eqs (rcu_dynticks_snap (cpu))); rdp->barrier_seq_snap = rcu_state.barrier_sequence; rdp->rcu_ofl_gp_seq = rcu_state.gp_seq;
@@ -4259,6 +4259,7 @@ rcu_boot_init_percpu_data(int cpu)int rcutree_prepare_cpu (unsigned int cpu) {unsigned long flags; +struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu);struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu);struct rcu_node *rnp =rcu_get_root ();
@@ -4267,7 +4268,7 @@ int rcutree_prepare_cpu(unsigned int cpu) rdp->qlen_last_fqs_check =0 ; rdp->n_force_qs_snap =READ_ONCE (rcu_state.n_force_qs); rdp->blimit = blimit; - rdp->dynticks_nesting =1 ;/* CPU not up, no tearing. */ + ct->dynticks_nesting =1 ;/* CPU not up, no tearing. */ raw_spin_unlock_rcu_node (rnp);/* irqs remain disabled. */ /*
kernel/rcu/tree.h +0 -1
@@ -191,7 +191,6 @@ struct rcu_data {/* 3) dynticks interface. */ int dynticks_snap;/* Per-GP tracking for dynticks. */ -long dynticks_nesting;/* Track process nesting level. */ long dynticks_nmi_nesting;/* Track irq/NMI nesting level. */ bool rcu_need_heavy_qs;/* GP old, so heavy quiescent state! */ bool rcu_urgent_qs;/* GP old need light quiescent state. */
kernel/rcu/tree_stall.h +1 -1
@@ -477,7 +477,7 @@ static void print_cpu_stall_info(int cpu)"!." [!delta], ticks_value, ticks_title,rcu_dynticks_snap (cpu) &0xfff , - rdp->dynticks_nesting, rdp->dynticks_nmi_nesting, +ct_dynticks_nesting_cpu (cpu), rdp->dynticks_nmi_nesting, rdp->softirq_snap,kstat_softirqs_cpu (RCU_SOFTIRQ , cpu),data_race (rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, rcuc_starved ? buf : "",
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 16/21 rcu/context_tracking: Move dynticks_nmi_nesting to context tracking
In reply to Frederic WeisbeckerThe RCU eqs tracking is going to be performed by the context tracking subsystem. The related nesting counters thus need to be moved to the context tracking structure.
--- include/linux/context_tracking_state.h | 15 ++++++++ kernel/context_tracking.c | 1 + kernel/rcu/rcu.h | 4 --- kernel/rcu/tree.c | 48 +++++++++++--------------- kernel/rcu/tree.h | 1 - kernel/rcu/tree_stall.h | 2 +- 6 files changed, 38 insertions(+), 33 deletions(-)
include/linux/context_tracking_state.h +15 -0
@@ -6,6 +6,9 @@#include <linux/static_key.h> #include <linux/context_tracking_irq.h> +/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ +#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) +struct context_tracking {#ifdef CONFIG_CONTEXT_TRACKING_USER /*
@@ -25,6 +28,7 @@ struct context_tracking {#endif atomic_t dynticks;/* Even value for idle, else odd. */ long dynticks_nesting;/* Track process nesting level. */ +long dynticks_nmi_nesting;/* Track irq/NMI nesting level. */ };#ifdef CONFIG_CONTEXT_TRACKING
@@ -57,6 +61,17 @@ static __always_inline long ct_dynticks_nesting_cpu(int cpu)struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu); return ct->dynticks_nesting; } + +static __always_inline long ct_dynticks_nmi_nesting (void ) +{ +return __this_cpu_read (context_tracking.dynticks_nmi_nesting); +} + +static __always_inline long ct_dynticks_nmi_nesting_cpu (int cpu) +{ +struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu); +return ct->dynticks_nmi_nesting; +}#endif #ifdef CONFIG_CONTEXT_TRACKING_USER
kernel/context_tracking.c +1 -0
@@ -250,6 +250,7 @@ void __init context_tracking_init(void)#endif /* #ifdef CONFIG_CONTEXT_TRACKING_USER */ DEFINE_PER_CPU(struct context_tracking, context_tracking) = { + .dynticks_nmi_nesting =DYNTICK_IRQ_NONIDLE , .dynticks_nesting =1 , .dynticks =ATOMIC_INIT (1 ), };
kernel/rcu/rcu.h +0 -4
@@ -12,10 +12,6 @@#include <trace/events/rcu.h> -/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ -#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) - -/* * Grace-period counter management. */
kernel/rcu/tree.c +21 -27
144 lines not on this page · raw message
kernel/rcu/tree.h +0 -1
@@ -191,7 +191,6 @@ struct rcu_data {/* 3) dynticks interface. */ int dynticks_snap;/* Per-GP tracking for dynticks. */ -long dynticks_nmi_nesting;/* Track irq/NMI nesting level. */ bool rcu_need_heavy_qs;/* GP old, so heavy quiescent state! */ bool rcu_urgent_qs;/* GP old need light quiescent state. */ bool rcu_forced_tick;/* Forced tick to provide QS. */
kernel/rcu/tree_stall.h +1 -1
@@ -477,7 +477,7 @@ static void print_cpu_stall_info(int cpu)"!." [!delta], ticks_value, ticks_title,rcu_dynticks_snap (cpu) &0xfff , -ct_dynticks_nesting_cpu (cpu), rdp->dynticks_nmi_nesting, +ct_dynticks_nesting_cpu (cpu),ct_dynticks_nmi_nesting_cpu (cpu), rdp->softirq_snap,kstat_softirqs_cpu (RCU_SOFTIRQ , cpu),data_race (rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart, rcuc_starved ? buf : "",
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 17/21 rcu/context-tracking: Move deferred nocb resched to context tracking
In reply to Frederic WeisbeckerTo prepare for migrating the RCU eqs accounting code to context tracking, split the last-resort deferred nocb resched from rcu_user_enter() and move it into a separate call from context tracking.
--- include/linux/rcutree.h | 6 ++++++ kernel/context_tracking.c | 8 ++++++++ kernel/rcu/tree.c | 15 ++------------- 3 files changed, 16 insertions(+), 13 deletions(-)
include/linux/rcutree.h +6 -0
@@ -76,4 +76,10 @@ int rcutree_dead_cpu(unsigned int cpu);int rcutree_dying_cpu (unsigned int cpu);void rcu_cpu_starting (unsigned int cpu); +#if defined(CONFIG_NO_HZ_FULL ) && (!defined(CONFIG_GENERIC_ENTRY ) || !defined(CONFIG_KVM_XFER_TO_GUEST_WORK )) +void rcu_irq_work_resched (void ); +#else +static inline void rcu_irq_work_resched (void ) { } +#endif +#endif /* __LINUX_RCUTREE_H */
kernel/context_tracking.c +8 -0
@@ -60,6 +60,8 @@ static __always_inline void context_tracking_recursion_exit(void)*/ void noinstr __ct_user_enter(enum ctx_state state) { +lockdep_assert_irqs_disabled (); +/* Kernel threads aren't supposed to go to userspace */ WARN_ON_ONCE (!current->mm);
@@ -81,6 +83,12 @@ void noinstr __ct_user_enter(enum ctx_state state)vtime_user_enter (current);instrumentation_end (); } +/* +* Other than generic entry implementation, we may be past the last +* rescheduling opportunity in the entry code. Trigger a self IPI +* that will fire and reschedule once we resume in user/guest mode. +*/ +rcu_irq_work_resched ();rcu_user_enter (); }/*
kernel/rcu/tree.c +2 -13
@@ -681,7 +681,7 @@ static DEFINE_PER_CPU(struct irq_work, late_wakeup_work) =* last resort is to fire a local irq_work that will trigger a reschedule once IRQs * get re-enabled again. */ -noinstr static void rcu_irq_work_resched(void) +noinstr void rcu_irq_work_resched(void) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data);
@@ -697,10 +697,7 @@ noinstr static void rcu_irq_work_resched(void) }instrumentation_end (); } - -#else -static inline void rcu_irq_work_resched (void ) { } -#endif +#endif /* #if !defined(CONFIG_GENERIC_ENTRY) || !defined(CONFIG_KVM_XFER_TO_GUEST_WORK) */ /** * rcu_user_enter - inform RCU that we are resuming userspace.
@@ -715,14 +712,6 @@ static inline void rcu_irq_work_resched(void) { }*/ noinstr void rcu_user_enter(void) { -lockdep_assert_irqs_disabled (); - -/* -* Other than generic entry implementation, we may be past the last -* rescheduling opportunity in the entry code. Trigger a self IPI -* that will fire and reschedule once we resume in user/guest mode. -*/ -rcu_irq_work_resched ();rcu_eqs_enter (true); }
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 18/21 rcu/context-tracking: Move RCU-dynticks internal functions to context_tracking
In reply to Frederic WeisbeckerMove the core RCU eqs/dynticks functions to context tracking so that we can later merge all that code within context tracking.
--- include/linux/context_tracking.h | 20 ++ include/linux/rcutree.h | 3 + kernel/context_tracking.c | 342 +++++++++++++++++++++++++++++++ kernel/rcu/tree.c | 328 +---------------------------- kernel/rcu/tree.h | 5 - kernel/rcu/tree_plugin.h | 38 +--- 6 files changed, 370 insertions(+), 366 deletions(-)
include/linux/context_tracking.h +20 -0
@@ -122,6 +122,26 @@ static inline void context_tracking_init(void) { }#ifdef CONFIG_CONTEXT_TRACKING extern void ct_idle_enter (void );extern void ct_idle_exit (void ); + +/* +* Is the current CPU in an extended quiescent state? +* +* No ordering, as we are sampling CPU-local information. +*/ +static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void) +{ +return !(arch_atomic_read (this_cpu_ptr (&context_tracking.dynticks)) &0x1 ); +} + +/* +* Increment the current CPU's context_tracking structure's ->dynticks field +* with ordering. Return the new value. +*/ +static __always_inline unsigned long rcu_dynticks_inc (int incby) +{ +return arch_atomic_add_return (incby,this_cpu_ptr (&context_tracking.dynticks)); +} +#else static inline void ct_idle_enter (void ) { }static inline void ct_idle_exit (void ) { }
include/linux/rcutree.h +3 -0
@@ -57,6 +57,9 @@ void rcu_irq_exit_check_preempt(void);static inline void rcu_irq_exit_check_preempt (void ) { }#endif +struct task_struct ; +void rcu_preempt_deferred_qs (struct task_struct *t); +void exit_rcu (void );void rcu_scheduler_starting (void );
kernel/context_tracking.c +342 -0
348 lines not on this page · raw message
kernel/rcu/tree.c +3 -325
371 lines not on this page · raw message
kernel/rcu/tree.h +0 -5
@@ -435,7 +435,6 @@ static void rcu_cpu_kthread_setup(unsigned int cpu);static void rcu_spawn_one_boost_kthread (struct rcu_node *rnp);static bool rcu_preempt_has_tasks (struct rcu_node *rnp);static bool rcu_preempt_need_deferred_qs (struct task_struct *t); -static void rcu_preempt_deferred_qs (struct task_struct *t);static void zero_cpu_stall_ticks (struct rcu_data *rdp);static struct swait_queue_head *rcu_nocb_gp_get (struct rcu_node *rnp);static void rcu_nocb_gp_cleanup (struct swait_queue_head *sq);
@@ -475,10 +474,6 @@ do { \static void rcu_bind_gp_kthread (void );static bool rcu_nohz_full_cpu (void ); -static void rcu_dynticks_task_enter (void ); -static void rcu_dynticks_task_exit (void ); -static void rcu_dynticks_task_trace_enter (void ); -static void rcu_dynticks_task_trace_exit (void );/* Forward declarations for tree_stall.h */ static void record_gp_stall_check_time (void );
kernel/rcu/tree_plugin.h +2 -36
@@ -595,7 +595,7 @@ static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t)* evaluate safety in terms of interrupt, softirq, and preemption * disabling. */ -static notrace void rcu_preempt_deferred_qs(struct task_struct *t) +notrace void rcu_preempt_deferred_qs(struct task_struct *t) {unsigned long flags;
@@ -935,7 +935,7 @@ static bool rcu_preempt_need_deferred_qs(struct task_struct *t)// period for a quiescent state from this CPU. Note that requests from // tasks are handled when removing the task from the blocked-tasks list // below. -static void rcu_preempt_deferred_qs (struct task_struct *t) +void rcu_preempt_deferred_qs (struct task_struct *t) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data);
@@ -1290,37 +1290,3 @@ static void rcu_bind_gp_kthread(void)return ;housekeeping_affine (current,HK_TYPE_RCU ); } - -/* Record the current task on dyntick-idle entry. */ -static __always_inline void rcu_dynticks_task_enter(void) -{ -#if defined(CONFIG_TASKS_RCU ) && defined(CONFIG_NO_HZ_FULL ) -WRITE_ONCE (current->rcu_tasks_idle_cpu,smp_processor_id ()); -#endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ -} - -/* Record no current task on dyntick-idle exit. */ -static __always_inline void rcu_dynticks_task_exit(void) -{ -#if defined(CONFIG_TASKS_RCU ) && defined(CONFIG_NO_HZ_FULL ) -WRITE_ONCE (current->rcu_tasks_idle_cpu,-1 ); -#endif /* #if defined(CONFIG_TASKS_RCU) && defined(CONFIG_NO_HZ_FULL) */ -} - -/* Turn on heavyweight RCU tasks trace readers on idle/user entry. */ -static __always_inline void rcu_dynticks_task_trace_enter(void) -{ -#ifdef CONFIG_TASKS_TRACE_RCU -if (IS_ENABLED (CONFIG_TASKS_TRACE_RCU_READ_MB )) - current->trc_reader_special.b.need_mb = true; -#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ -} - -/* Turn off heavyweight RCU tasks trace readers on idle/user exit. */ -static __always_inline void rcu_dynticks_task_trace_exit(void) -{ -#ifdef CONFIG_TASKS_TRACE_RCU -if (IS_ENABLED (CONFIG_TASKS_TRACE_RCU_READ_MB )) - current->trc_reader_special.b.need_mb = false; -#endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ -}
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 19/21 rcu/context-tracking: Remove unused and/or unecessary middle functions
In reply to Frederic WeisbeckerSome eqs functions are now only used internally by context tracking, so their public declarations can be removed.
Also middle functions such as rcu_user_*() and rcu_idle_*() which now directly call to rcu_eqs_enter() and rcu_eqs_exit() can be wiped out as well.
--- Documentation/RCU/stallwarn.rst | 2 +- include/linux/hardirq.h | 8 -- include/linux/rcupdate.h | 8 -- include/linux/rcutiny.h | 2 - include/linux/rcutree.h | 2 - kernel/context_tracking.c | 147 +++++++++++--------------------- 6 files changed, 50 insertions(+), 119 deletions(-)
Documentation/RCU/stallwarn.rst +1 -1
@@ -97,7 +97,7 @@ warnings: which will include additional debugging information. - A low-level kernel issue that either fails to invoke one of the - variants of rcu_user_enter(), rcu_user_exit(), ct_idle_enter(), + variants of rcu_eqs_enter(true), rcu_eqs_exit(true), ct_idle_enter(), ct_idle_exit(), ct_irq_enter(), or ct_irq_exit() on the one hand, or that invokes one of them too many times on the other. Historically, the most frequent issue has been an omission
include/linux/hardirq.h +0 -8
@@ -92,14 +92,6 @@ void irq_exit_rcu(void);#define arch_nmi_exit () do { } while (0)#endif -#ifdef CONFIG_TINY_RCU -static inline void rcu_nmi_enter (void ) { } -static inline void rcu_nmi_exit (void ) { } -#else -extern void rcu_nmi_enter (void ); -extern void rcu_nmi_exit (void ); -#endif -/* * NMI vs Tracing * --------------
include/linux/rcupdate.h +0 -8
@@ -105,14 +105,6 @@ static inline void rcu_sysrq_start(void) { }static inline void rcu_sysrq_end (void ) { }#endif /* #else #ifdef CONFIG_RCU_STALL_COMMON */ -#ifdef CONFIG_NO_HZ_FULL -void rcu_user_enter (void ); -void rcu_user_exit (void ); -#else -static inline void rcu_user_enter (void ) { } -static inline void rcu_user_exit (void ) { } -#endif /* CONFIG_NO_HZ_FULL */ -#ifdef CONFIG_RCU_NOCB_CPU void rcu_init_nohz (void );int rcu_nocb_cpu_offload (int cpu);
include/linux/rcutiny.h +0 -2
@@ -86,8 +86,6 @@ static inline int rcu_needs_cpu(void)static inline void rcu_virt_note_context_switch (int cpu) { }static inline void rcu_cpu_stall_reset (void ) { }static inline int rcu_jiffies_till_stall_check (void ) {return 21 *HZ ; } -static inline void rcu_idle_enter (void ) { } -static inline void rcu_idle_exit (void ) { }static inline void rcu_irq_exit_check_preempt (void ) { }#define rcu_is_idle_cpu (cpu) \ (is_idle_task(current) && !in_nmi() && !in_hardirq() && !in_serving_softirq())
include/linux/rcutree.h +0 -2
@@ -47,8 +47,6 @@ unsigned long start_poll_synchronize_rcu(void);bool poll_state_synchronize_rcu (unsigned long oldstate);void cond_synchronize_rcu (unsigned long oldstate); -void rcu_idle_enter (void ); -void rcu_idle_exit (void );bool rcu_is_idle_cpu (int cpu);#ifdef CONFIG_PROVE_RCU
kernel/context_tracking.c +49 -98
210 lines not on this page · raw message
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 20/21 context_tracking: Convert state to atomic_t
In reply to Frederic WeisbeckerContext tracking's state and dynticks counter are going to be merged in a single field so that both updates can happen atomically and at the same time. Prepare for that with converting the state into an atomic_t.
--- include/linux/context_tracking.h | 18 ++---------- include/linux/context_tracking_state.h | 40 ++++++++++++++++++++++---- kernel/context_tracking.c | 15 ++++++---- 3 files changed, 46 insertions(+), 27 deletions(-)
include/linux/context_tracking.h +3 -15
@@ -56,7 +56,7 @@ static inline enum ctx_state exception_enter(void) !context_tracking_enabled())return 0 ; - prev_ctx =this_cpu_read (context_tracking.state); + prev_ctx =__ct_state ();if (prev_ctx !=CONTEXT_KERNEL )ct_user_exit (prev_ctx);
@@ -86,26 +86,14 @@ static __always_inline void context_tracking_guest_exit(void)__ct_user_exit (CONTEXT_GUEST ); } -/** -* ct_state() - return the current context tracking state if known -* -* Returns the current cpu's context tracking state if context tracking -* is enabled. If context tracking is disabled, returns -* CONTEXT_DISABLED. This should be used primarily for debugging. -*/ -static __always_inline enum ctx_state ct_state(void) -{ -return context_tracking_enabled () ? -this_cpu_read (context_tracking.state) :CONTEXT_DISABLED ; -}#else static inline void user_enter (void ) { }static inline void user_exit (void ) { }static inline void user_enter_irqoff (void ) { }static inline void user_exit_irqoff (void ) { } -static inline enum ctx_state exception_enter (void ) {return 0 ; } +static inline int exception_enter (void ) {return 0 ; }static inline void exception_exit (enum ctx_state prev_ctx) { } -static inline enum ctx_state ct_state (void ) {return CONTEXT_DISABLED ; } +static inline int ct_state (void ) {return -1 ; } static __always_inline bool context_tracking_guest_enter(void) { return false; }static inline void context_tracking_guest_exit (void ) { }
include/linux/context_tracking_state.h +34 -6
@@ -9,6 +9,13 @@/* Offset to allow distinguishing irq vs. task-based idle entry/exit. */ #define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1) +enum ctx_state { +CONTEXT_DISABLED =-1 ,/* returned by ct_state() if unknown */ +CONTEXT_KERNEL =0 , +CONTEXT_USER , +CONTEXT_GUEST , +}; +struct context_tracking {#ifdef CONFIG_CONTEXT_TRACKING_USER /*
@@ -19,12 +26,7 @@ struct context_tracking {*/ bool active;int recursion; -enum ctx_state { -CONTEXT_DISABLED =-1 ,/* returned by ct_state() if unknown */ -CONTEXT_KERNEL =0 , -CONTEXT_USER , -CONTEXT_GUEST , - } state; +atomic_t state;#endif atomic_t dynticks;/* Even value for idle, else odd. */ long dynticks_nesting;/* Track process nesting level. */
@@ -34,6 +36,11 @@ struct context_tracking {#ifdef CONFIG_CONTEXT_TRACKING DECLARE_PER_CPU(struct context_tracking, context_tracking); +static __always_inline int __ct_state(void) +{ +return atomic_read (this_cpu_ptr (&context_tracking.state)); +} + static __always_inline int ct_dynticks(void) {return atomic_read (this_cpu_ptr (&context_tracking.dynticks));
@@ -92,6 +99,27 @@ static inline bool context_tracking_enabled_this_cpu(void) return context_tracking_enabled() && __this_cpu_read(context_tracking.active); } +/** +* ct_state() - return the current context tracking state if known +* +* Returns the current cpu's context tracking state if context tracking +* is enabled. If context tracking is disabled, returns +* CONTEXT_DISABLED. This should be used primarily for debugging. +*/ +static __always_inline int ct_state(void) +{ +int ret; + +if (!context_tracking_enabled ()) +return CONTEXT_DISABLED ; + +preempt_disable (); + ret =__ct_state (); +preempt_enable (); + +return ret; +} +#else static inline bool context_tracking_enabled (void ) {return false; }static inline bool context_tracking_enabled_cpu (int cpu) {return false; }
kernel/context_tracking.c +9 -6
@@ -328,6 +328,7 @@ static __always_inline void context_tracking_recursion_exit(void)*/ void noinstr __ct_user_enter(enum ctx_state state) { +struct context_tracking *ct =this_cpu_ptr (&context_tracking);lockdep_assert_irqs_disabled ();/* Kernel threads aren't supposed to go to userspace */
@@ -336,8 +337,8 @@ void noinstr __ct_user_enter(enum ctx_state state)if (!context_tracking_recursion_enter ())return ; -if (__this_cpu_read (context_tracking.state) != state) { -if (__this_cpu_read (context_tracking.active)) { +if (__ct_state () != state) { +if (ct->active) {/* * At this stage, only low level arch entry code remains and * then we'll run in userspace. We can assume there won't be
@@ -378,7 +379,7 @@ void noinstr __ct_user_enter(enum ctx_state state)* OTOH we can spare the calls to vtime and RCU when context_tracking.active * is false because we know that CPU is not tickless. */ -__this_cpu_write (context_tracking.state, state); +atomic_set (&ct->state, state); }context_tracking_recursion_exit (); }
@@ -439,11 +440,13 @@ NOKPROBE_SYMBOL(user_enter_callable);*/ void noinstr __ct_user_exit(enum ctx_state state) { +struct context_tracking *ct =this_cpu_ptr (&context_tracking); +if (!context_tracking_recursion_enter ())return ; -if (__this_cpu_read (context_tracking.state) == state) { -if (__this_cpu_read (context_tracking.active)) { +if (__ct_state () == state) { +if (ct->active) {/* * Exit RCU idle mode while entering the kernel because it can * run a RCU read side critical section anytime.
@@ -456,7 +459,7 @@ void noinstr __ct_user_exit(enum ctx_state state)instrumentation_end (); } } -__this_cpu_write (context_tracking.state,CONTEXT_KERNEL ); +atomic_set (&ct->state,CONTEXT_KERNEL ); }context_tracking_recursion_exit (); }
Signature · 1 line
2.25.1
Re: [PATCH 20/21] context_tracking: Convert state to atomic_t
On Tue, 2022-05-03 at 12:00 +0200, Frederic Weisbecker wrote:
[...]
Quoted 11 lines
include/linux/context_tracking_state.h +11
-0+/** +* ct_state() - return the current context tracking state if known +* +* Returns the current cpu's context tracking state if context tracking +* is enabled. If context tracking is disabled, returns +* CONTEXT_DISABLED. This should be used primarily for debugging. +*/ +static __always_inline int ct_state(void) +{ +int ret; +
include/linux/context_tracking_state.h static inline bool context_tracking_enabled_this_cpu(void) · line 122
Quoted 10 lines
include/linux/context_tracking_state.h +10
-0+if (!context_tracking_enabled ()) +return CONTEXT_DISABLED ; + +preempt_disable (); + ret =__ct_state (); +preempt_enable (); + +return ret; +} +
I can't see any use for this function with preemption enabled. You can't trust the data due to CPU migration and it could be a source of bugs in the future. Wouldn't it make more sense to move the burden into the callers? They all DTRT, plus, this_cpu_ptr() will spew warnings if someone shows up and doesn't comply.
Regards,
Signature · 1 line
Nicolás Sáenz
Frederic Weisbecker
maintainerRe: [PATCH 20/21] context_tracking: Convert state to atomic_t
Quoted 22 lines · nicolas saenz julienne
On Tue, 2022-05-03 at 12:00 +0200, Frederic Weisbecker wrote:
[...]
Quoted 18 lines
include/linux/context_tracking_state.h +18
-0+/** +* ct_state() - return the current context tracking state if known +* +* Returns the current cpu's context tracking state if context tracking +* is enabled. If context tracking is disabled, returns +* CONTEXT_DISABLED. This should be used primarily for debugging. +*/ +static __always_inline int ct_state(void) +{ +int ret; + +if (!context_tracking_enabled ()) +return CONTEXT_DISABLED ; + +preempt_disable (); + ret =__ct_state (); +preempt_enable (); +
Quoted 8 lines
include/linux/context_tracking_state.h static inline bool context_tracking_enabled_this_cpu(void) · line 122
Quoted 3 lines
include/linux/context_tracking_state.h +3
-0+return ret; +} +I can't see any use for this function with preemption enabled. You can't trust the data due to CPU migration and it could be a source of bugs in the future. Wouldn't it make more sense to move the burden into the callers? They all DTRT, plus,
this_cpu_ptr()will spew warnings if someone shows up and doesn't comply.
I believe syscall_exit_to_user_mode_prepare() has preemption enabled. Then it's ok if we are scheduled away right before the check, it still applies on any CPU.
Re: [PATCH 20/21] context_tracking: Convert state to atomic_t
Quoted 32 lines · Frederic Weisbecker
Quoted 22 lines · nicolas saenz julienne
On Tue, 2022-05-03 at 12:00 +0200, Frederic Weisbecker wrote:
[...]
Quoted 18 lines
include/linux/context_tracking_state.h +18
-0+/** +* ct_state() - return the current context tracking state if known +* +* Returns the current cpu's context tracking state if context tracking +* is enabled. If context tracking is disabled, returns +* CONTEXT_DISABLED. This should be used primarily for debugging. +*/ +static __always_inline int ct_state(void) +{ +int ret; + +if (!context_tracking_enabled ()) +return CONTEXT_DISABLED ; + +preempt_disable (); + ret =__ct_state (); +preempt_enable (); +Quoted 8 lines
include/linux/context_tracking_state.h static inline bool context_tracking_enabled_this_cpu(void) · line 122
Quoted 3 lines
include/linux/context_tracking_state.h +3
-0+return ret; +} +I can't see any use for this function with preemption enabled. You can't trust the data due to CPU migration and it could be a source of bugs in the future. Wouldn't it make more sense to move the burden into the callers? They all DTRT, plus,
this_cpu_ptr()will spew warnings if someone shows up and doesn't comply.
Quoted 3 lines
I believe
syscall_exit_to_user_mode_prepare()has preemption enabled. Then it's ok if we are scheduled away right before the check, it still applies on any CPU.
Fair enough.
Signature · 1 line
Nicolás Sáenz
Frederic Weisbecker
maintainerPATCH 21/21 rcu/context_tracking: Merge dynticks counter and context tracking states
In reply to Frederic WeisbeckerUpdating the context tracking state and the RCU dynticks counter atomically in a single operation is a first step towards improving CPU isolation. This makes the context tracking state updates fully ordered and therefore allow for later enhancements such as postponing some work while a task is running isolated in userspace until it ever comes back to the kernel.
The state field becomes divided in two parts:
1) Two Lower bits for context tracking state:
CONTEXT_KERNEL = 0
CONTEXT_IDLE = 1,
CONTEXT_USER = 2,
CONTEXT_GUEST = 3,
2) Higher bits for RCU eqs dynticks counting:
RCU_DYNTICKS_IDX = 4
The dynticks counting is always incremented by this value. (state & RCU_DYNTICKS_IDX) means we are NOT in an extended quiescent state. This makes the chance for a collision more likely between two RCU dynticks snapshots but wrapping up 28 bits of eqs dynticks increments still takes some bad luck (also rdp.dynticks_snap could be converted from int to long?)
Some RCU eqs functions have been renamed to better reflect their broader scope that now include context tracking state.
--- include/linux/context_tracking.h | 8 +-- include/linux/context_tracking_state.h | 27 +++++--- kernel/context_tracking.c | 88 +++++++++++++------------- kernel/rcu/tree.c | 13 ++-- kernel/rcu/tree_stall.h | 4 +- 5 files changed, 74 insertions(+), 66 deletions(-)
include/linux/context_tracking.h +4 -4
@@ -118,16 +118,16 @@ extern void ct_idle_exit(void);*/ static __always_inline bool rcu_dynticks_curr_cpu_in_eqs(void) { -return !(arch_atomic_read (this_cpu_ptr (&context_tracking.dynticks)) &0x1 ); +return !(arch_atomic_read (this_cpu_ptr (&context_tracking.state)) &RCU_DYNTICKS_IDX ); }/* -* Increment the current CPU's context_tracking structure's ->dynticks field +* Increment the current CPU's context_tracking structure's ->state field * with ordering. Return the new value. */ -static __always_inline unsigned long rcu_dynticks_inc (int incby) +static __always_inline unsigned long ct_state_inc (int incby) { -return arch_atomic_add_return (incby,this_cpu_ptr (&context_tracking.dynticks)); +return arch_atomic_add_return (incby,this_cpu_ptr (&context_tracking.state)); }#else
include/linux/context_tracking_state.h +17 -10
@@ -10,12 +10,20 @@#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1)enum ctx_state { -CONTEXT_DISABLED =-1 ,/* returned by ct_state() if unknown */ -CONTEXT_KERNEL =0 , -CONTEXT_USER , -CONTEXT_GUEST , +CONTEXT_DISABLED =-1 ,/* returned by ct_state() if unknown */ +CONTEXT_KERNEL =0 , +CONTEXT_IDLE =1 , +CONTEXT_USER =2 , +CONTEXT_GUEST =3 , +CONTEXT_MAX =4 , }; +/* Even value for idle, else odd. */ +#define RCU_DYNTICKS_IDX CONTEXT_MAX + +#define CT_STATE_MASK (CONTEXT_MAX - 1) +#define CT_DYNTICKS_MASK (~CT_STATE_MASK) +struct context_tracking {#ifdef CONFIG_CONTEXT_TRACKING_USER /*
@@ -26,9 +34,8 @@ struct context_tracking {*/ bool active;int recursion; +#endif atomic_t state; -#endif -atomic_t dynticks;/* Even value for idle, else odd. */ long dynticks_nesting;/* Track process nesting level. */ long dynticks_nmi_nesting;/* Track irq/NMI nesting level. */ };
@@ -38,24 +45,24 @@ DECLARE_PER_CPU(struct context_tracking, context_tracking); static __always_inline int __ct_state(void) { -return atomic_read (this_cpu_ptr (&context_tracking.state)); +return atomic_read (this_cpu_ptr (&context_tracking.state)) &CT_STATE_MASK ; } static __always_inline int ct_dynticks(void) { -return atomic_read (this_cpu_ptr (&context_tracking.dynticks)); +return atomic_read (this_cpu_ptr (&context_tracking.state)) &CT_DYNTICKS_MASK ; } static __always_inline int ct_dynticks_cpu(int cpu) {struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu); -return atomic_read (&ct->dynticks); +return atomic_read (&ct->state) &CT_DYNTICKS_MASK ; } static __always_inline int ct_dynticks_cpu_acquire(int cpu) {struct context_tracking *ct =per_cpu_ptr (&context_tracking, cpu); -return atomic_read_acquire (&ct->state); +return atomic_read_acquire (&ct->state) &CT_DYNTICKS_MASK ; }static __always_inline long ct_dynticks_nesting (void )
kernel/context_tracking.c +45 -43
215 lines not on this page · raw message
kernel/rcu/tree.c +6 -7
@@ -272,9 +272,9 @@ void rcu_softirq_qs(void)*/ static void rcu_dynticks_eqs_online (void ) { -if (ct_dynticks () &0x1 ) +if (ct_dynticks () &RCU_DYNTICKS_IDX )return ; -rcu_dynticks_inc (1 ); +ct_state_inc (RCU_DYNTICKS_IDX ); }/*
@@ -293,7 +293,7 @@ static int rcu_dynticks_snap(int cpu)*/ static bool rcu_dynticks_in_eqs (int snap) { -return !(snap &0x1 ); +return !(snap &RCU_DYNTICKS_IDX ); }/* Return true if the specified CPU is currently idle from an RCU viewpoint. */
@@ -321,8 +321,7 @@ bool rcu_dynticks_zero_in_eqs(int cpu, int *vp)int snap;// If not quiescent, force back to earlier extended quiescent state. - snap =ct_dynticks_cpu (cpu) & ~0x1 ; - + snap =ct_dynticks_cpu (cpu) & ~RCU_DYNTICKS_IDX ;smp_rmb ();// Order ->dynticks and *vp reads. if (READ_ONCE (*vp))return false;// Non-zero, so report failure;
@@ -348,9 +347,9 @@ notrace void rcu_momentary_dyntick_idle(void)int seq;raw_cpu_write (rcu_data.rcu_need_heavy_qs, false); - seq =rcu_dynticks_inc (2 ); + seq =ct_state_inc (2 *RCU_DYNTICKS_IDX );/* It is illegal to call this from idle state. */ -WARN_ON_ONCE (!(seq &0x1 )); +WARN_ON_ONCE (!(seq &RCU_DYNTICKS_IDX ));rcu_preempt_deferred_qs (current); }EXPORT_SYMBOL_GPL (rcu_momentary_dyntick_idle);
kernel/rcu/tree_stall.h +2 -2
@@ -467,7 +467,7 @@ static void print_cpu_stall_info(int cpu) rcuc_starved =rcu_is_rcuc_kthread_starving (rdp, &j);if (rcuc_starved)sprintf (buf," rcuc=%ld jiffies(starved)" , j); - pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%03x/%ld/%#lx softirq=%u/%u fqs=%ld%s%s\n", + pr_err("\t%d-%c%c%c%c: (%lu %s) idle=%04x/%ld/%#lx softirq=%u/%u fqs=%ld%s%s\n", cpu, "O."[!!cpu_online(cpu)], "o."[!!(rdp->grpmask & rdp->mynode->qsmaskinit)],
@@ -476,7 +476,7 @@ static void print_cpu_stall_info(int cpu) rdp->rcu_iw_pending ? (int )min (delta,9UL ) +'0' :"!." [!delta], ticks_value, ticks_title, -rcu_dynticks_snap (cpu) &0xfff , +rcu_dynticks_snap (cpu) &0xffff ,ct_dynticks_nesting_cpu (cpu),ct_dynticks_nmi_nesting_cpu (cpu), rdp->softirq_snap,kstat_softirqs_cpu (RCU_SOFTIRQ , cpu), data_race(rcu_state.n_force_qs) - rcu_state.n_force_qs_gpstart,
Signature · 1 line
2.25.1
Re: [PATCH 21/21] rcu/context_tracking: Merge dynticks counter and context tracking states
On Tue, 2022-05-03 at 12:00 +0200, Frederic Weisbecker wrote:
[...]
Quoted 15 lines
include/linux/context_tracking_state.h +3
-4@@ -10,12 +10,20 @@#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1)enum ctx_state { -CONTEXT_DISABLED =-1 ,/* returned by ct_state() if unknown */ -CONTEXT_KERNEL =0 , -CONTEXT_USER , -CONTEXT_GUEST , +CONTEXT_DISABLED =-1 ,/* returned by ct_state() if unknown */ +CONTEXT_KERNEL =0 , +CONTEXT_IDLE =1 ,
include/linux/context_tracking_state.h line 25
Quoted 10 lines
include/linux/context_tracking_state.h +8
-0+CONTEXT_USER =2 , +CONTEXT_GUEST =3 , + CONTEXT_MAX = 4, }; +/* Even value for idle, else odd. */ +#define RCU_DYNTICKS_IDX CONTEXT_MAX + +#define CT_STATE_MASK (CONTEXT_MAX - 1) +#define CT_DYNTICKS_MASK (~CT_STATE_MASK)
Nit: These macros only work as long as we don't introduce new contexts. It's maybe worth a comment.
Regards,
Signature · 1 line
Nicolás Sáenz
Frederic Weisbecker
maintainerRe: [PATCH 21/21] rcu/context_tracking: Merge dynticks counter and context tracking states
Quoted 24 lines · nicolas saenz julienne
On Tue, 2022-05-03 at 12:00 +0200, Frederic Weisbecker wrote:
[...]
Quoted 20 lines
include/linux/context_tracking_state.h +6
-4@@ -10,12 +10,20 @@#define DYNTICK_IRQ_NONIDLE ((LONG_MAX / 2) + 1)enum ctx_state { -CONTEXT_DISABLED =-1 ,/* returned by ct_state() if unknown */ -CONTEXT_KERNEL =0 , -CONTEXT_USER , -CONTEXT_GUEST , +CONTEXT_DISABLED =-1 ,/* returned by ct_state() if unknown */ +CONTEXT_KERNEL =0 , +CONTEXT_IDLE =1 , +CONTEXT_USER =2 , +CONTEXT_GUEST =3 , +CONTEXT_MAX =4 , };
Quoted 8 lines
include/linux/context_tracking_state.h line 25
Quoted 5 lines
include/linux/context_tracking_state.h +5
-0+/* Even value for idle, else odd. */ +#define RCU_DYNTICKS_IDX CONTEXT_MAX + +#define CT_STATE_MASK (CONTEXT_MAX - 1) +#define CT_DYNTICKS_MASK (~CT_STATE_MASK)Nit: These macros only work as long as we don't introduce new contexts. It's maybe worth a comment.
Ah good point, forgot that.
Thanks!
Quoted 18 lines · Frederic Weisbecker
This mixes up the RCU dynticks counter and the context tracking state updates into a single atomic instruction. This may serve several purposes:
1) Improve CPU isolation with deferring some disturbances until sensitive userspace workload completes and goes to the kernel. This can take several forms, for example
smp_call_function_housekeeping()oron_each_housekeeping_cpu()to enqueue and execute work on all housekeeping CPUs. Then an atomic operation onct->statecan defer the work onnohz_fullCPUs until they run in kernel (or IPI them if they are in kernel mode), see this proposal by Peter:https://lore.kernel.org/all/[email protected]/#r2) Unearth sysidle (https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/commit/?h=sysidle.2017.05.11a&id=fe5ac724d81a3c7803e60c2232718f212f3f38d4)
This feature allowed to shutdown the tick on the last housekeeping
CPU once the rest of the system is fully idle. We needed some proper
fully ordered context tracking for that.
Quoted 5 lines
Inspired by Peterz: https://lore.kernel.org/all/[email protected]
Passed 100 hours of TREE01.
---
For the whole series:
Thanks!
Signature · 1 line
Nicolás Sáenz
Frederic Weisbecker
maintainerQuoted 21 lines · Nicolas Saenz Julienne
Quoted 20 lines · Frederic Weisbecker
This mixes up the RCU dynticks counter and the context tracking state updates into a single atomic instruction. This may serve several purposes:
1) Improve CPU isolation with deferring some disturbances until sensitive userspace workload completes and goes to the kernel. This can take several forms, for example
smp_call_function_housekeeping()oron_each_housekeeping_cpu()to enqueue and execute work on all housekeeping CPUs. Then an atomic operation onct->statecan defer the work onnohz_fullCPUs until they run in kernel (or IPI them if they are in kernel mode), see this proposal by Peter:https://lore.kernel.org/all/[email protected]/#r2) Unearth sysidle (https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git/commit/?h=sysidle.2017.05.11a&id=fe5ac724d81a3c7803e60c2232718f212f3f38d4)
This feature allowed to shutdown the tick on the last housekeeping
CPU once the rest of the system is fully idle. We needed some proper
fully ordered context tracking for that.Inspired by Peterz: https://lore.kernel.org/all/[email protected]
Quoted 8 lines
Quoted 3 lines
Passed 100 hours of TREE01.
---
For the whole series:
Reviewed-by Nicolas Saenz Julienne <[email protected]>
I hesitated to apply these but the series has changed a little, although the fundamental bits remain.
I'll just wait for a confirmation from you on this v3.
Thanks!