Miscellaneous fixes for v6.1
Paul E. McKenney
maintainercover Miscellaneous fixes for v6.1
Hello!
This series contains miscellaneous fixes:
1. Fix rcu_read_unlock_strict() strict QS reporting, courtesy
of Zqiang.
2. Update rcu_preempt_deferred_qs() comments for !PREEMPT kernels,
courtesy of Zqiang.
3. Add QS check in rcu_exp_handler() for non-preemptible kernels,
courtesy of Zqiang.
4. Make tiny RCU support leak callbacks for debug-object errors,
courtesy of Zqiang.
5. Document reason for rcu_all_qs() call to preempt_disable().
6. Update rcu_access_pointer() header for
rcu_dereference_protected().
7. Try trigger_single_cpu_backtrace(cpu) in dump_cpu_task(),
courtesy of Zhen Lei.
8. Show the registers of 'current' in dump_cpu_task(), courtesy of
Zhen Lei.
9. Avoid triggering strict-GP irq-work when RCU is idle, courtesy
of Zqiang.
10. Exclude outgoing CPU when it is the last to leave.
Thanx, Paul
------------------------------------------------------------------------
b/include/linux/rcupdate.h | 18 +++++++++++++----- b/kernel/rcu/tiny.c | 17 ++++++++++++++++- b/kernel/rcu/tree_exp.h | 4 +++- b/kernel/rcu/tree_plugin.h | 1 + b/kernel/rcu/tree_stall.h | 5 ++--- b/kernel/sched/core.c | 3 +++ b/kernel/smp.c | 3 +-- kernel/rcu/tree_plugin.h | 21 ++++++++++++++------- kernel/sched/core.c | 11 +++++++++++ 9 files changed, 64 insertions(+), 19 deletions(-)
Paul E. McKenney
maintainerPATCH 1/10 rcu: Fix rcu_read_unlock_strict() strict QS reporting
From: Zqiang <[email protected]>
Kernels built with CONFIG_PREEMPT=n and CONFIG_RCU_STRICT_GRACE_PERIOD=y report the quiescent state directly from the outermost rcu_read_unlock(). However, the current CPU's rcu_data structure's ->cpu_no_qs.b.norm might still be set, in which case rcu_report_qs_rdp() will exit early, thus failing to report quiescent state.
This commit therefore causes rcu_read_unlock_strict() to clear CPU's rcu_data structure's ->cpu_no_qs.b.norm field before invoking rcu_report_qs_rdp().
--- kernel/rcu/tree_plugin.h | 1 + 1 file changed, 1 insertion(+)
kernel/rcu/tree_plugin.h +1 -0
@@ -824,6 +824,7 @@ void rcu_read_unlock_strict(void)if (irqs_disabled () ||preempt_count () || !rcu_state.gp_kthread)return ; rdp =this_cpu_ptr (&rcu_data); + rdp->cpu_no_qs.b.norm = false;rcu_report_qs_rdp (rdp);udelay (rcu_unlock_delay); }
Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerPATCH 2/10 rcu: Update rcu_preempt_deferred_qs() comments for !PREEMPT kernels
In reply to Paul E. McKenneyFrom: Zqiang <[email protected]>
In non-premptible kernels, tasks never do context switches within RCU read-side critical sections. Therefore, in such kernels, each leaf rcu_node structure's ->blkd_tasks list will always be empty. The comment on the non-preemptible version of rcu_preempt_deferred_qs() confuses this point, so this commit therefore fixes it.
--- kernel/rcu/tree_plugin.h | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
kernel/rcu/tree_plugin.h +7 -4
@@ -932,10 +932,13 @@ static notrace bool rcu_preempt_need_deferred_qs(struct task_struct *t)return false; } -// Except that we do need to respond to a request by an expedited grace -// period for a quiescent state from this CPU. Note that requests from -// tasks are handled when removing the task from the blocked-tasks list -// below. +// Except that we do need to respond to a request by an expedited +// grace period for a quiescent state from this CPU. Note that in +// non-preemptible kernels, there can be no context switches within RCU +// read-side critical sections, which in turn means that the leaf rcu_node +// structure's blocked-tasks list is always empty. is therefore no need to +// actually check it. Instead, a quiescent state from this CPU suffices, +// and this function is only called from such a quiescent state. notrace void rcu_preempt_deferred_qs(struct task_struct *t) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data);
Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerPATCH 3/10 rcu: Add QS check in rcu_exp_handler() for non-preemptible kernels
In reply to Paul E. McKenneyFrom: Zqiang <[email protected]>
Kernels built with CONFIG_PREEMPTION=n and CONFIG_PREEMPT_COUNT=y maintain preempt_count() state. Because such kernels map __rcu_read_lock() and __rcu_read_unlock() to preempt_disable() and preempt_enable(), respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU version of the rcu_exp_handler() IPI handler function to use preempt_count() to detect quiescent states.
This preempt_count() usage might seem to risk failures due to use of implicit RCU readers in portions of the kernel under #ifndef CONFIG_PREEMPTION, except that rcu_core() already disallows such implicit RCU readers. The moral of this story is that you must use explicit read-side markings such as rcu_read_lock() or preempt_disable() even if the code knows that this kernel does not support preemption.
This commit therefore adds a preempt_count()-based check for a quiescent state in the !CONFIG_PREEMPT_RCU version of the rcu_exp_handler() function for kernels built with CONFIG_PREEMPT_COUNT=y, reporting an immediate quiescent state when the interrupted code had both preemption and softirqs enabled.
This change results in about a 2% reduction in expedited grace-period latency in kernels built with both CONFIG_PREEMPT_RCU=n and CONFIG_PREEMPT_COUNT=y.
--- kernel/rcu/tree_exp.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
kernel/rcu/tree_exp.h +3 -1
@@ -828,11 +828,13 @@ static void rcu_exp_handler(void *unused) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; +bool preempt_bh_enabled = !(preempt_count () & (PREEMPT_MASK |SOFTIRQ_MASK ));if (!(READ_ONCE (rnp->expmask) & rdp->grpmask) ||__this_cpu_read (rcu_data.cpu_no_qs.b.exp))return ; -if (rcu_is_cpu_rrupt_from_idle ()) { +if (rcu_is_cpu_rrupt_from_idle () || + (IS_ENABLED (CONFIG_PREEMPT_COUNT ) && preempt_bh_enabled)) {rcu_report_exp_rdp (this_cpu_ptr (&rcu_data));return ; }
Signature · 1 line
2.31.1.189.g2e36527f23
Frederic Weisbecker
maintainerRe: [PATCH rcu 03/10] rcu: Add QS check in rcu_exp_handler() for non-preemptible kernels
Quoted 38 lines · Paul E. McKenney
From: Zqiang <[email protected]>
Kernels built with
CONFIG_PREEMPTION=nandCONFIG_PREEMPT_COUNT=ymaintainpreempt_count()state. Because such kernels map__rcu_read_lock()and__rcu_read_unlock()topreempt_disable()andpreempt_enable(), respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU version of thercu_exp_handler()IPI handler function to usepreempt_count()to detect quiescent states.This
preempt_count()usage might seem to risk failures due to use of implicit RCU readers in portions of the kernel under #ifndefCONFIG_PREEMPTION, except thatrcu_core()already disallows such implicit RCU readers. The moral of this story is that you must use explicit read-side markings such asrcu_read_lock()orpreempt_disable()even if the code knows that this kernel does not support preemption.This commit therefore adds a preempt_count()-based check for a quiescent state in the !CONFIG_PREEMPT_RCU version of the
rcu_exp_handler()function for kernels built withCONFIG_PREEMPT_COUNT=y, reporting an immediate quiescent state when the interrupted code had both preemption and softirqs enabled.This change results in about a 2% reduction in expedited grace-period latency in kernels built with both
CONFIG_PREEMPT_RCU=nandCONFIG_PREEMPT_COUNT=y.Signed-off-by Zqiang <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>--- kernel/rcu/tree_exp.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)kernel/rcu/tree_exp.h +0
-0
kernel/rcu/tree_exp.h static void rcu_exp_handler(void *unused) · line 832
Quoted 5 lines
kernel/rcu/tree_exp.h +1
-0@@ -828,11 +828,13 @@ static void rcu_exp_handler(void *unused) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; +bool preempt_bh_enabled = !(preempt_count () & (PREEMPT_MASK |SOFTIRQ_MASK ));
I don't know if nested hardirqs still exist. I only heard old rumours about broken drivers. Should we take care of them?
Also are we sure that all callers of flush_smp_call_function_queue() are QS?
Let's see we know that rcu_exp_handler() can either be executed from:
* hardirqs
Or from process context, expected to be RCU QS states at least in idle as the comment above flush_smp_call_function_queue() in idle says (but I'd rather check all the in-process callers before stating all of them are in QS)
* idle (in which case preemption is disabled unfortunately so the current test won't help)
* stop_machine
_ When CPU is dead and out of RCU (rcutree_dead_cpu() called)
so that should be a QS.
_ When CPU is migrating (is it a QS then?)
If we check further that all non-IRQ callers of flush_smp_call_function_queue() are always quiescent states then we could deduce that !in_hardirq() means we are in a quiescent state, whether preemption is disabled or not.
In any case for the current patch, perhaps a more robust test against nested hardirqs would be:
unsigned long cnt = preempt_count();
bool preempt_bh_enabled = (!cnt || cnt == HARDIRQ_OFFSET)
Thanks.
Quoted 13 lines
kernel/rcu/tree_exp.h +2
-1if (!(READ_ONCE (rnp->expmask) & rdp->grpmask) ||__this_cpu_read (rcu_data.cpu_no_qs.b.exp))return ; -if (rcu_is_cpu_rrupt_from_idle ()) { +if (rcu_is_cpu_rrupt_from_idle () || + (IS_ENABLED (CONFIG_PREEMPT_COUNT ) && preempt_bh_enabled)) {rcu_report_exp_rdp (this_cpu_ptr (&rcu_data));return ; }Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerRe: [PATCH rcu 03/10] rcu: Add QS check in rcu_exp_handler() for non-preemptible kernels
Quoted 34 lines · Frederic Weisbecker
Quoted 33 lines · Paul E. McKenney
From: Zqiang <[email protected]>
Kernels built with
CONFIG_PREEMPTION=nandCONFIG_PREEMPT_COUNT=ymaintainpreempt_count()state. Because such kernels map__rcu_read_lock()and__rcu_read_unlock()topreempt_disable()andpreempt_enable(), respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU version of thercu_exp_handler()IPI handler function to usepreempt_count()to detect quiescent states.This
preempt_count()usage might seem to risk failures due to use of implicit RCU readers in portions of the kernel under #ifndefCONFIG_PREEMPTION, except thatrcu_core()already disallows such implicit RCU readers. The moral of this story is that you must use explicit read-side markings such asrcu_read_lock()orpreempt_disable()even if the code knows that this kernel does not support preemption.This commit therefore adds a preempt_count()-based check for a quiescent state in the !CONFIG_PREEMPT_RCU version of the
rcu_exp_handler()function for kernels built withCONFIG_PREEMPT_COUNT=y, reporting an immediate quiescent state when the interrupted code had both preemption and softirqs enabled.This change results in about a 2% reduction in expedited grace-period latency in kernels built with both
CONFIG_PREEMPT_RCU=nandCONFIG_PREEMPT_COUNT=y.Signed-off-by Zqiang <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>--- kernel/rcu/tree_exp.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Quoted 13 lines
kernel/rcu/tree_exp.h static void rcu_exp_handler(void *unused) · line 832
Quoted 10 lines
kernel/rcu/tree_exp.h +1
-0@@ -828,11 +828,13 @@ static void rcu_exp_handler(void *unused) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; +bool preempt_bh_enabled = !(preempt_count () & (PREEMPT_MASK |SOFTIRQ_MASK ));I don't know if nested hardirqs still exist. I only heard old rumours about broken drivers. Should we take care of them?
Last I checked, certain tracing scenarios from irq handlers looked to RCU like nested irq handlers. Given that, does your more robust approach below work correctly?
Thanx, Paul
Quoted 44 lines
Also are we sure that all callers of
flush_smp_call_function_queue()are QS?Let's see we know that
rcu_exp_handler()can either be executed from:* hardirqs
Or from process context, expected to be RCU QS states at least in idle as the comment above
flush_smp_call_function_queue()in idle says (but I'd rather check all the in-process callers before stating all of them are in QS)* idle (in which case preemption is disabled unfortunately so the current test won't help)
*stop_machine
_ When CPU is dead and out of RCU (rcutree_dead_cpu()called)
so that should be a QS.
_ When CPU is migrating (is it a QS then?)If we check further that all non-IRQ callers of
flush_smp_call_function_queue()are always quiescent states then we could deduce that !in_hardirq() means we are in a quiescent state, whether preemption is disabled or not.In any case for the current patch, perhaps a more robust test against nested hardirqs would be:
unsigned long cnt =
preempt_count();
boolpreempt_bh_enabled= (!cnt || cnt ==HARDIRQ_OFFSET)Thanks.
Quoted 13 lines
kernel/rcu/tree_exp.h +2
-1if (!(READ_ONCE (rnp->expmask) & rdp->grpmask) ||__this_cpu_read (rcu_data.cpu_no_qs.b.exp))return ; -if (rcu_is_cpu_rrupt_from_idle ()) { +if (rcu_is_cpu_rrupt_from_idle () || + (IS_ENABLED (CONFIG_PREEMPT_COUNT ) && preempt_bh_enabled)) {rcu_report_exp_rdp (this_cpu_ptr (&rcu_data));return ; }Signature · 1 line
2.31.1.189.g2e36527f23
Frederic Weisbecker
maintainerRe: [PATCH rcu 03/10] rcu: Add QS check in rcu_exp_handler() for non-preemptible kernels
Quoted 49 lines · Paul E. McKenney
Quoted 34 lines · Frederic Weisbecker
Quoted 33 lines · Paul E. McKenney
From: Zqiang <[email protected]>
Kernels built with
CONFIG_PREEMPTION=nandCONFIG_PREEMPT_COUNT=ymaintainpreempt_count()state. Because such kernels map__rcu_read_lock()and__rcu_read_unlock()topreempt_disable()andpreempt_enable(), respectively, this allows the expedited grace period's !CONFIG_PREEMPT_RCU version of thercu_exp_handler()IPI handler function to usepreempt_count()to detect quiescent states.This
preempt_count()usage might seem to risk failures due to use of implicit RCU readers in portions of the kernel under #ifndefCONFIG_PREEMPTION, except thatrcu_core()already disallows such implicit RCU readers. The moral of this story is that you must use explicit read-side markings such asrcu_read_lock()orpreempt_disable()even if the code knows that this kernel does not support preemption.This commit therefore adds a preempt_count()-based check for a quiescent state in the !CONFIG_PREEMPT_RCU version of the
rcu_exp_handler()function for kernels built withCONFIG_PREEMPT_COUNT=y, reporting an immediate quiescent state when the interrupted code had both preemption and softirqs enabled.This change results in about a 2% reduction in expedited grace-period latency in kernels built with both
CONFIG_PREEMPT_RCU=nandCONFIG_PREEMPT_COUNT=y.Signed-off-by Zqiang <[email protected]>Signed-off-by Paul E. McKenney <[email protected]>--- kernel/rcu/tree_exp.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)Quoted 13 lines
kernel/rcu/tree_exp.h static void rcu_exp_handler(void *unused) · line 832
Quoted 10 lines
kernel/rcu/tree_exp.h +1
-0@@ -828,11 +828,13 @@ static void rcu_exp_handler(void *unused) {struct rcu_data *rdp =this_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; +bool preempt_bh_enabled = !(preempt_count () & (PREEMPT_MASK |SOFTIRQ_MASK ));I don't know if nested hardirqs still exist. I only heard old rumours about broken drivers. Should we take care of them?
Quoted 3 lines
Last I checked, certain tracing scenarios from irq handlers looked to RCU like nested irq handlers. Given that, does your more robust approach below work correctly?
I haven't observed that but in any case, the check I propose is more strict than the one on this patch. So in the worst case it's a QS not reported if a nested interrupt is detected.
Thanks.
Paul E. McKenney
maintainerPATCH 4/10 rcu: Make tiny RCU support leak callbacks for debug-object errors
In reply to Paul E. McKenneyFrom: Zqiang <[email protected]>
Currently, only Tree RCU leaks callbacks setting when it detects a duplicate call_rcu(). This commit causes Tiny RCU to also leak callbacks in this situation.
Because this is Tiny RCU, kernel size is important:
1. CONFIG_TINY_RCU=y and CONFIG_DEBUG_OBJECTS_RCU_HEAD=n (Production kernel)
Original:
text data bss dec hex filename
26290663 20159823 15212544 61663030 3ace736 vmlinux
With this commit:
text data bss dec hex filename
26290663 20159823 15212544 61663030 3ace736 vmlinux
2. CONFIG_TINY_RCU=y and CONFIG_DEBUG_OBJECTS_RCU_HEAD=y (Debugging kernel)
Original:
text data bss dec hex filename
26291319 20160143 15212544 61664006 3aceb06 vmlinux
With this commit:
text data bss dec hex filename
26291319 20160431 15212544 61664294 3acec26 vmlinux
These results show that the kernel size is unchanged for production kernels, as desired.
--- kernel/rcu/tiny.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
kernel/rcu/tiny.c +16 -1
@@ -158,6 +158,10 @@ void synchronize_rcu(void) }EXPORT_SYMBOL_GPL (synchronize_rcu); +static void tiny_rcu_leak_callback (struct rcu_head *rhp) +{ +} +/* * Post an RCU callback to be invoked after the end of an RCU grace * period. But since we have but one CPU, that would be after any
@@ -165,9 +169,20 @@ EXPORT_SYMBOL_GPL(synchronize_rcu);*/ void call_rcu (struct rcu_head *head,rcu_callback_t func) { +static atomic_t doublefrees;unsigned long flags; -debug_rcu_head_queue (head); +if (debug_rcu_head_queue (head)) { +if (atomic_inc_return (&doublefrees) <4 ) { +pr_err ("%s(): Double-freed CB %p->%pS()!!! " , __func__, head, head->func); +mem_dump_obj (head); + } + +if (!__is_kvfree_rcu_offset ((unsigned long )head->func)) +WRITE_ONCE (head->func, tiny_rcu_leak_callback); +return ; + } + head->func = func; head->next =NULL ;
Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerPATCH 5/10 rcu: Document reason for rcu_all_qs() call to preempt_disable()
In reply to Paul E. McKenneyGiven that rcu_all_qs() is in non-preemptible kernels, why on earth should it invoke preempt_disable()? This commit adds the reason, which is to work nicely with debugging enabled in CONFIG_PREEMPT_COUNT=y kernels.
--- kernel/rcu/tree_plugin.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +1 -1
@@ -870,7 +870,7 @@ void rcu_all_qs(void)if (!raw_cpu_read (rcu_data.rcu_urgent_qs))return ; -preempt_disable (); +preempt_disable ();// For CONFIG_PREEMPT_COUNT=y kernels /* Load rcu_urgent_qs before other flags. */ if (!smp_load_acquire (this_cpu_ptr (&rcu_data.rcu_urgent_qs))) {preempt_enable ();
Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerPATCH 6/10 rcu: Update rcu_access_pointer() header for rcu_dereference_protected()
In reply to Paul E. McKenneyThe rcu_access_pointer() docbook header correctly notes that it may be used during post-grace-period teardown. However, it is usually better to use rcu_dereference_protected() for this purpose. This commit therefore calls out this preferred usage.
--- include/linux/rcupdate.h | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
include/linux/rcupdate.h +13 -5
@@ -496,13 +496,21 @@ do { \* against NULL. Although rcu_access_pointer() may also be used in cases * where update-side locks prevent the value of the pointer from changing, * you should instead use rcu_dereference_protected() for this use case. +* Within an RCU read-side critical section, there is little reason to +* use rcu_access_pointer(). +* +* It is usually best to test the rcu_access_pointer() return value +* directly in order to avoid accidental dereferences being introduced +* by later inattentive changes. In other words, assigning the +* rcu_access_pointer() return value to a local variable results in an +* accident waiting to happen. * * It is also permissible to use rcu_access_pointer() when read-side -* access to the pointer was removed at least one grace period ago, as -* is the case in the context of the RCU callback that is freeing up -* the data, or after a synchronize_rcu() returns. This can be useful -* when tearing down multi-linked structures after a grace period -* has elapsed. +* access to the pointer was removed at least one grace period ago, as is +* the case in the context of the RCU callback that is freeing up the data, +* or after a synchronize_rcu() returns. This can be useful when tearing +* down multi-linked structures after a grace period has elapsed. However, +* rcu_dereference_protected() is normally preferred for this use case. */ #define rcu_access_pointer (p) __rcu_access_pointer((p), __UNIQUE_ID(rcu), __rcu)
Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerPATCH 7/10 sched/debug: Try trigger_single_cpu_backtrace(cpu) in dump_cpu_task()
In reply to Paul E. McKenneyFrom: Zhen Lei <[email protected]>
The trigger_all_cpu_backtrace() function attempts to send an NMI to the target CPU, which usually provides much better stack traces than the dump_cpu_task() function's approach of dumping that stack from some other CPU. So much so that most calls to dump_cpu_task() only happen after a call to trigger_all_cpu_backtrace() has failed. And the exception to this rule really should attempt to use trigger_all_cpu_backtrace() first.
Therefore, move the trigger_all_cpu_backtrace() invocation into dump_cpu_task().
--- kernel/rcu/tree_stall.h | 5 ++--- kernel/sched/core.c | 3 +++ kernel/smp.c | 3 +-- 3 files changed, 6 insertions(+), 5 deletions(-)
kernel/rcu/tree_stall.h +2 -3
@@ -368,7 +368,7 @@ static void rcu_dump_cpu_stacks(void)if (rnp->qsmask &leaf_node_cpu_bit (rnp, cpu)) {if (cpu_is_offline (cpu))pr_err ("Offline CPU %d blocking current GP.\n" , cpu); -else if (!trigger_single_cpu_backtrace (cpu)) +else dump_cpu_task (cpu); }raw_spin_unlock_irqrestore_rcu_node (rnp, flags);
@@ -511,8 +511,7 @@ static void rcu_check_gp_kthread_starvation(void)pr_err ("RCU GP kthread last ran on offline CPU %d.\n" , cpu); }else {pr_err ("Stack dump where RCU GP kthread last ran:\n" ); -if (!trigger_single_cpu_backtrace (cpu)) -dump_cpu_task (cpu); +dump_cpu_task (cpu); } }wake_up_process (gpk);
kernel/sched/core.c +3 -0
@@ -11183,6 +11183,9 @@ struct cgroup_subsys cpu_cgrp_subsys = {void dump_cpu_task (int cpu) { +if (trigger_single_cpu_backtrace (cpu)) +return ; +pr_info ("Task dump for CPU %d:\n" , cpu);sched_show_task (cpu_curr (cpu)); }
kernel/smp.c +1 -2
@@ -370,8 +370,7 @@ static bool csd_lock_wait_toolong(struct __call_single_data *csd, u64 ts0, u64 *if (cpu >=0 ) {if (static_branch_unlikely (&csdlock_debug_extended))csd_lock_print_extended (csd, cpu); -if (!trigger_single_cpu_backtrace (cpu)) -dump_cpu_task (cpu); +dump_cpu_task (cpu);if (!cpu_cur_csd) {pr_alert ("csd: Re-sending CSD lock (#%d) IPI from CPU#%02d to CPU#%02d\n" , *bug_id,raw_smp_processor_id (), cpu);arch_send_call_function_single_ipi (cpu);
Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerPATCH 8/10 sched/debug: Show the registers of 'current' in dump_cpu_task()
In reply to Paul E. McKenneyFrom: Zhen Lei <[email protected]>
The dump_cpu_task() function does not print registers on architectures that do not support NMIs. However, registers can be useful for debugging. Fortunately, in the case where dump_cpu_task() is invoked from an interrupt handler and is dumping the current CPU's stack, the get_irq_regs() function can be used to get the registers.
Therefore, this commit makes dump_cpu_task() check to see if it is being asked to dump the current CPU's stack from within an interrupt handler, and, if so, it uses the get_irq_regs() function to obtain the registers. On systems that do support NMIs, this commit has the further advantage of avoiding a self-NMI in this case.
This is an example of rcu self-detected stall on arm64, which does not support NMIs:
[ 27.501721] rcu: INFO: rcu_preempt self-detected stall on CPU [ 27.502238] rcu: 0-....: (1250 ticks this GP) idle=4f7/1/0x4000000000000000 softirq=2594/2594 fqs=619 [ 27.502632] (t=1251 jiffies g=2989 q=29 ncpus=4) [ 27.503845] CPU: 0 PID: 306 Comm: test0 Not tainted 5.19.0-rc7-00009-g1c1a6c29ff99-dirty #46 [ 27.504732] Hardware name: linux,dummy-virt (DT) [ 27.504947] pstate: 20000005 (nzCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 27.504998] pc : arch_counter_read+0x18/0x24 [ 27.505301] lr : arch_counter_read+0x18/0x24 [ 27.505328] sp : ffff80000b29bdf0 [ 27.505345] x29: ffff80000b29bdf0 x28: 0000000000000000 x27: 0000000000000000 [ 27.505475] x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 [ 27.505553] x23: 0000000000001f40 x22: ffff800009849c48 x21: 000000065f871ae0 [ 27.505627] x20: 00000000000025ec x19: ffff80000a6eb300 x18: ffffffffffffffff [ 27.505654] x17: 0000000000000001 x16: 0000000000000000 x15: ffff80000a6d0296 [ 27.505681] x14: ffffffffffffffff x13: ffff80000a29bc18 x12: 0000000000000426 [ 27.505709] x11: 0000000000000162 x10: ffff80000a2f3c18 x9 : ffff80000a29bc18 [ 27.505736] x8 : 00000000ffffefff x7 : ffff80000a2f3c18 x6 : 00000000759bd013 [ 27.505761] x5 : 01ffffffffffffff x4 : 0002dc6c00000000 x3 : 0000000000000017 [ 27.505787] x2 : 00000000000025ec x1 : ffff80000b29bdf0 x0 : 0000000075a30653 [ 27.505937] Call trace: [ 27.506002] arch_counter_read+0x18/0x24 [ 27.506171] ktime_get+0x48/0xa0 [ 27.506207] test_task+0x70/0xf0 [ 27.506227] kthread+0x10c/0x110 [ 27.506243] ret_from_fork+0x10/0x20
This is a marked improvement over the old output:
[ 27.944550] rcu: INFO: rcu_preempt self-detected stall on CPU [ 27.944980] rcu: 0-....: (1249 ticks this GP) idle=cbb/1/0x4000000000000000 softirq=2610/2610 fqs=614 [ 27.945407] (t=1251 jiffies g=2681 q=28 ncpus=4) [ 27.945731] Task dump for CPU 0: [ 27.945844] task:test0 state:R running task stack: 0 pid: 306 ppid: 2 flags:0x0000000a [ 27.946073] Call trace: [ 27.946151] dump_backtrace.part.0+0xc8/0xd4 [ 27.946378] show_stack+0x18/0x70 [ 27.946405] sched_show_task+0x150/0x180 [ 27.946427] dump_cpu_task+0x44/0x54 [ 27.947193] rcu_dump_cpu_stacks+0xec/0x130 [ 27.947212] rcu_sched_clock_irq+0xb18/0xef0 [ 27.947231] update_process_times+0x68/0xac [ 27.947248] tick_sched_handle+0x34/0x60 [ 27.947266] tick_sched_timer+0x4c/0xa4 [ 27.947281] __hrtimer_run_queues+0x178/0x360 [ 27.947295] hrtimer_interrupt+0xe8/0x244 [ 27.947309] arch_timer_handler_virt+0x38/0x4c [ 27.947326] handle_percpu_devid_irq+0x88/0x230 [ 27.947342] generic_handle_domain_irq+0x2c/0x44 [ 27.947357] gic_handle_irq+0x44/0xc4 [ 27.947376] call_on_irq_stack+0x2c/0x54 [ 27.947415] do_interrupt_handler+0x80/0x94 [ 27.947431] el1_interrupt+0x34/0x70 [ 27.947447] el1h_64_irq_handler+0x18/0x24 [ 27.947462] el1h_64_irq+0x64/0x68 <--- the above backtrace is worthless [ 27.947474] arch_counter_read+0x18/0x24 [ 27.947487] ktime_get+0x48/0xa0 [ 27.947501] test_task+0x70/0xf0 [ 27.947520] kthread+0x10c/0x110 [ 27.947538] ret_from_fork+0x10/0x20
--- kernel/sched/core.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
kernel/sched/core.c +11 -0
@@ -73,6 +73,7 @@#include <uapi/linux/sched/types.h> +#include <asm/irq_regs.h> #include <asm/switch_to.h> #include <asm/tlb.h>
@@ -11183,6 +11184,16 @@ struct cgroup_subsys cpu_cgrp_subsys = {void dump_cpu_task (int cpu) { +if (cpu ==smp_processor_id () &&in_hardirq ()) { +struct pt_regs *regs; + + regs =get_irq_regs (); +if (regs) { +show_regs (regs); +return ; + } + } +if (trigger_single_cpu_backtrace (cpu))return ;
Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerPATCH 9/10 rcu: Avoid triggering strict-GP irq-work when RCU is idle
In reply to Paul E. McKenneyFrom: Zqiang <[email protected]>
Kernels built with PREEMPT_RCU=y and RCU_STRICT_GRACE_PERIOD=y trigger irq-work from rcu_read_unlock(), and the resulting irq-work handler invokes rcu_preempt_deferred_qs_handle(). The point of this triggering is to force grace periods to end quickly in order to give tools like KASAN a better chance of detecting RCU usage bugs such as leaking RCU-protected pointers out of an RCU read-side critical section.
However, this irq-work triggering is unconditional. This works, but there is no point in doing this irq-work unless the current grace period is waiting on the running CPU or task, which is not the common case. After all, in the common case there are many rcu_read_unlock() calls per CPU per grace period.
This commit therefore triggers the irq-work only when the current grace period is waiting on the running CPU or task.
This change was tested as follows on a four-CPU system:
echo rcu_preempt_deferred_qs_handler > /sys/kernel/debug/tracing/set_ftrace_filter echo 1 > /sys/kernel/debug/tracing/function_profile_enabled
insmod rcutorture.ko
sleep 20
rmmod rcutorture.ko
echo 0 > /sys/kernel/debug/tracing/function_profile_enabled
echo > /sys/kernel/debug/tracing/set_ftrace_filter
This procedure produces results in this per-CPU set of files:
/sys/kernel/debug/tracing/trace_stat/function*
Sample output from one of these files is as follows:
Function Hit Time Avg s^2 -------- --- ---- --- --- rcu_preempt_deferred_qs_handle 838746 182650.3 us 0.217 us 0.004 us
The baseline sum of the "Hit" values (the number of calls to this function) was 3,319,015. With this commit, that sum was 1,140,359, for a 2.9x reduction. The worst-case variance across the CPUs was less than 25%, so this large effect size is statistically significant.
The raw data is available in the Link: URL.
--- kernel/rcu/tree_plugin.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +2 -1
@@ -641,7 +641,8 @@ static void rcu_read_unlock_special(struct task_struct *t) expboost = (t->rcu_blocked_node &&READ_ONCE (t->rcu_blocked_node->exp_tasks)) || (rdp->grpmask &READ_ONCE (rnp->expmask)) || -IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD ) || + (IS_ENABLED (CONFIG_RCU_STRICT_GRACE_PERIOD ) && + ((rdp->grpmask &READ_ONCE (rnp->qsmask)) || t->rcu_blocked_node)) || (IS_ENABLED (CONFIG_RCU_BOOST ) && irqs_were_disabled && t->rcu_blocked_node);// Need to defer quiescent state until everything is enabled.
Signature · 1 line
2.31.1.189.g2e36527f23
Paul E. McKenney
maintainerPATCH 10/10 rcu: Exclude outgoing CPU when it is the last to leave
In reply to Paul E. McKenneyThe rcu_boost_kthread_setaffinity() function removes the outgoing CPU from the set_cpus_allowed() mask for the corresponding leaf rcu_node structure's rcub priority-boosting kthread. Except that if the outgoing CPU will leave that structure without any online CPUs, the mask is set to the housekeeping CPU mask from housekeeping_cpumask(). Which is fine unless the outgoing CPU happens to be a housekeeping CPU.
This commit therefore removes the outgoing CPU from the housekeeping mask. This would of course be problematic if the outgoing CPU was the last online housekeeping CPU, but in that case you are in a world of hurt anyway. If someone comes up with a valid use case for a system needing all the housekeeping CPUs to be offline, further adjustments can be made.
--- kernel/rcu/tree_plugin.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +4 -1
@@ -1243,8 +1243,11 @@ static void rcu_boost_kthread_setaffinity(struct rcu_node *rnp, int outgoingcpu) cpu != outgoingcpu)cpumask_set_cpu (cpu, cm);cpumask_and (cm, cm,housekeeping_cpumask (HK_TYPE_RCU )); -if (cpumask_empty (cm)) +if (cpumask_empty (cm)) {cpumask_copy (cm,housekeeping_cpumask (HK_TYPE_RCU )); +if (outgoingcpu >=0 ) +cpumask_clear_cpu (outgoingcpu, cm); + }set_cpus_allowed_ptr (t, cm);mutex_unlock (&rnp->boost_kthread_mutex);free_cpumask_var (cm);
Signature · 1 line
2.31.1.189.g2e36527f23