RCU tasks updates for v6.12
Subsystems (3)
Neeraj Upadhyay
maintainercover RCU tasks updates for v6.12
1. Remove RCU Tasks Rude asynchronous APIs from rcu-updaters.sh,
courtesy of Paul E. McKenney.
2. Remove RCU Tasks Rude asynchronous APIs, courtesy of
Paul E. McKenney.
3. Stop testing RCU Tasks Rude asynchronous APIs, courtesy of
Paul E. McKenney.
4. Stop testing RCU Tasks Rude asynchronous APIs, courtesy of
Paul E. McKenney.
5. Remove RCU Tasks Rude asynchronous APIs, courtesy of
Paul E. McKenney.
6. Fix access non-existent percpu rtpcp variable in rcu_tasks_need_gpcb(),
courtesy of Zqiang.
7. Check processor-ID assumptions, courtesy of Paul E. McKenney.
8. Update rtp->tasks_gp_seq comment, courtesy of Paul E. McKenney.
9. Provide rcu_barrier_cb_is_done() to check rcu_barrier() CBs,
courtesy of Paul E. McKenney.
10. Mark callbacks not currently participating in barrier
operation, courtesy of Paul E. McKenney.
11. Add detailed grace-period and barrier diagnostics, courtesy of
Paul E. McKenney.
12. Add rcu_barrier_tasks*() start time to diagnostics, courtesy of
Paul E. McKenney.
- Neeraj
------------------------------------------------------------------------
.../RCU/Design/Requirements/Requirements.rst | 3 +- Documentation/RCU/checklist.rst | 61 +++-- Documentation/RCU/whatisRCU.rst | 2 +- .../admin-guide/kernel-parameters.txt | 8 - include/linux/rcupdate.h | 5 +- kernel/rcu/rcu.h | 5 + kernel/rcu/rcuscale.c | 2 - kernel/rcu/rcutorture.c | 8 - kernel/rcu/tasks.h | 212 ++++++++++++------ tools/rcu/rcu-updaters.sh | 2 - 10 files changed, 180 insertions(+), 128 deletions(-)
[email protected]
maintainerPATCH 1/12 tools/rcu: Remove RCU Tasks Rude asynchronous APIs from rcu-updaters.sh
From: "Paul E. McKenney" <[email protected]>
The call_rcu_tasks_rude() and rcu_barrier_tasks_rude() APIs are no longer. This commit therefore removes them from the rcu-updaters.sh script.
--- tools/rcu/rcu-updaters.sh | 2 -- 1 file changed, 2 deletions(-)
tools/rcu/rcu-updaters.sh +0 -2
@@ -21,12 +21,10 @@ fi bpftrace -e 'kprobe:kvfree_call_rcu, kprobe:call_rcu, kprobe:call_rcu_tasks, - kprobe:call_rcu_tasks_rude, kprobe:call_rcu_tasks_trace, kprobe:call_srcu, kprobe:rcu_barrier, kprobe:rcu_barrier_tasks, - kprobe:rcu_barrier_tasks_rude, kprobe:rcu_barrier_tasks_trace, kprobe:srcu_barrier, kprobe:synchronize_rcu,
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 2/12 doc: Remove RCU Tasks Rude asynchronous APIs
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
The call_rcu_tasks_rude() and rcu_barrier_tasks_rude() APIs are no longer. This commit therefore removes them from the documentation.
--- .../RCU/Design/Requirements/Requirements.rst | 3 +- Documentation/RCU/checklist.rst | 61 +++++++++---------- Documentation/RCU/whatisRCU.rst | 2 +- .../admin-guide/kernel-parameters.txt | 8 --- 4 files changed, 30 insertions(+), 44 deletions(-)
Documentation/RCU/Design/Requirements/Requirements.rst +1 -2
@@ -2649,8 +2649,7 @@ those that are idle from RCU's perspective) and then Tasks Rude RCU can be removed from the kernel. The tasks-rude-RCU API is also reader-marking-free and thus quite compact, -consisting of call_rcu_tasks_rude(), synchronize_rcu_tasks_rude(), -and rcu_barrier_tasks_rude(). +consisting solely of synchronize_rcu_tasks_rude(). Tasks Trace RCU ~~~~~~~~~~~~~~~
Documentation/RCU/checklist.rst +28 -33
103 lines not on this page · raw message
Documentation/RCU/whatisRCU.rst +1 -1
@@ -1103,7 +1103,7 @@ RCU-Tasks-Rude:: Critical sections Grace period Barrier - N/A call_rcu_tasks_rude rcu_barrier_tasks_rude + N/A N/A synchronize_rcu_tasks_rude
Documentation/admin-guide/kernel-parameters.txt +0 -8
@@ -5572,14 +5572,6 @@ of zero will disable batching. Batching is always disabled for synchronize_rcu_tasks(). - rcupdate.rcu_tasks_rude_lazy_ms= [KNL] - Set timeout in milliseconds RCU Tasks - Rude asynchronous callback batching for - call_rcu_tasks_rude(). A negative value - will take the default. A value of zero will - disable batching. Batching is always disabled - for synchronize_rcu_tasks_rude(). - rcupdate.rcu_tasks_trace_lazy_ms= [KNL] Set timeout in milliseconds RCU Tasks Trace asynchronous callback batching for
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 3/12 rcutorture: Stop testing RCU Tasks Rude asynchronous APIs
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
The call_rcu_tasks_rude() and rcu_barrier_tasks_rude() APIs are currently unused. Furthermore, the idea is to get rid of RCU Tasks Rude entirely once all architectures have their deep-idle and entry/exit code correctly marked as inline or noinstr. As a first step towards this goal, this commit therefore removes these two functions from rcutorture testing.
--- kernel/rcu/rcutorture.c | 8 -------- 1 file changed, 8 deletions(-)
kernel/rcu/rcutorture.c +0 -8
@@ -915,11 +915,6 @@ static struct rcu_torture_ops tasks_ops = {* Definitions for rude RCU-tasks torture testing. */ -static void rcu_tasks_rude_torture_deferred_free (struct rcu_torture *p) -{ -call_rcu_tasks_rude (&p->rtort_rcu, rcu_torture_cb); -} -static struct rcu_torture_ops tasks_rude_ops = { .ttype =RCU_TASKS_RUDE_FLAVOR , .init = rcu_sync_torture_init,
@@ -927,11 +922,8 @@ static struct rcu_torture_ops tasks_rude_ops = { .read_delay = rcu_read_delay, /* just reuse rcu's version. */ .readunlock = rcu_torture_read_unlock_trivial, .get_gp_seq = rcu_no_completed, - .deferred_free = rcu_tasks_rude_torture_deferred_free, .sync = synchronize_rcu_tasks_rude, .exp_sync = synchronize_rcu_tasks_rude, - .call = call_rcu_tasks_rude, - .cb_barrier = rcu_barrier_tasks_rude, .gp_kthread_dbg = show_rcu_tasks_rude_gp_kthread, .get_gp_data = rcu_tasks_rude_get_gp_data, .cbflood_max = 50000,
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 4/12 rcuscale: Stop testing RCU Tasks Rude asynchronous APIs
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
The call_rcu_tasks_rude() and rcu_barrier_tasks_rude() APIs are currently unused. Furthermore, the idea is to get rid of RCU Tasks Rude entirely once all architectures have their deep-idle and entry/exit code correctly marked as inline or noinstr. As a step towards this goal, this commit therefore removes these two functions from rcuscale testing.
--- kernel/rcu/rcuscale.c | 2 -- 1 file changed, 2 deletions(-)
kernel/rcu/rcuscale.c +0 -2
@@ -333,8 +333,6 @@ static struct rcu_scale_ops tasks_rude_ops = { .readunlock = tasks_rude_scale_read_unlock, .get_gp_seq = rcu_no_completed, .gp_diff = rcu_seq_diff, - .async = call_rcu_tasks_rude, - .gp_barrier = rcu_barrier_tasks_rude, .sync = synchronize_rcu_tasks_rude, .exp_sync = synchronize_rcu_tasks_rude, .rso_gp_kthread = get_rcu_tasks_rude_gp_kthread,
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 5/12 rcu-tasks: Remove RCU Tasks Rude asynchronous APIs
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
The call_rcu_tasks_rude() and rcu_barrier_tasks_rude() APIs are currently unused. This commit therefore removes their definitions and boot-time self-tests.
--- include/linux/rcupdate.h | 2 -- kernel/rcu/tasks.h | 55 +++++++++++++--------------------------- 2 files changed, 17 insertions(+), 40 deletions(-)
include/linux/rcupdate.h +0 -2
@@ -37,7 +37,6 @@/* Exported common interfaces */ void call_rcu (struct rcu_head *head,rcu_callback_t func);void rcu_barrier_tasks (void ); -void rcu_barrier_tasks_rude (void );void synchronize_rcu (void );struct rcu_gp_oldstate ;
@@ -202,7 +201,6 @@ do { \ } while (0)# ifdef CONFIG_TASKS_RUDE_RCU -void call_rcu_tasks_rude (struct rcu_head *head,rcu_callback_t func);void synchronize_rcu_tasks_rude (void );# endif
kernel/rcu/tasks.h +17 -38
125 lines not on this page · raw message
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 6/12 rcu-tasks: Fix access non-existent percpu rtpcp variable in rcu_tasks_need_gpcb()
In reply to Neeraj UpadhyayFrom: Zqiang <[email protected]>
For kernels built with CONFIG_FORCE_NR_CPUS=y, the nr_cpu_ids is defined as NR_CPUS instead of the number of possible cpus, this will cause the following system panic:
smpboot: Allowing 4 CPUs, 0 hotplug CPUs
...setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:512 nr_node_ids:1
...
BUG: unable to handle page fault for address: ffffffff9911c8c8 Oops: 0000 [#1] PREEMPT SMP PTI CPU: 0 PID: 15 Comm: rcu_tasks_trace Tainted: G W 6.6.21 #1 5dc7acf91a5e8e9ac9dcfc35bee0245691283ea6 RIP: 0010:rcu_tasks_need_gpcb+0x25d/0x2c0 RSP: 0018:ffffa371c00a3e60 EFLAGS: 00010082 CR2: ffffffff9911c8c8 CR3: 000000040fa20005 CR4: 00000000001706f0 Call Trace: <TASK> ? __die+0x23/0x80 ? page_fault_oops+0xa4/0x180 ? exc_page_fault+0x152/0x180 ? asm_exc_page_fault+0x26/0x40 ? rcu_tasks_need_gpcb+0x25d/0x2c0 ? __pfx_rcu_tasks_kthread+0x40/0x40 rcu_tasks_one_gp+0x69/0x180 rcu_tasks_kthread+0x94/0xc0 kthread+0xe8/0x140 ? __pfx_kthread+0x40/0x40 ret_from_fork+0x34/0x80 ? __pfx_kthread+0x40/0x40 ret_from_fork_asm+0x1b/0x80 </TASK>
Considering that there may be holes in the CPU numbers, use the maximum possible cpu number, instead of nr_cpu_ids, for configuring enqueue and dequeue limits.
[ neeraj.upadhyay: Fix htmldocs build error reported by Stephen Rothwell ]
--- kernel/rcu/tasks.h | 82 ++++++++++++++++++++++++++++++---------------- 1 file changed, 53 insertions(+), 29 deletions(-)
kernel/rcu/tasks.h +53 -29
@@ -34,6 +34,7 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp);* @rtp_blkd_tasks: List of tasks blocked as readers. * @rtp_exit_list: List of tasks in the latter portion of do_exit(). * @cpu: CPU number corresponding to this entry. +* @index: Index of this CPU in rtpcp_array of the rcu_tasks structure. * @rtpp: Pointer to the rcu_tasks structure. */ struct rcu_tasks_percpu {
@@ -49,6 +50,7 @@ struct rcu_tasks_percpu {struct list_head rtp_blkd_tasks;struct list_head rtp_exit_list;int cpu; +int index;struct rcu_tasks *rtpp; };
@@ -76,6 +78,7 @@ struct rcu_tasks_percpu {* @call_func: This flavor's call_rcu()-equivalent function. * @wait_state: Task state for synchronous grace-period waits (default TASK_UNINTERRUPTIBLE). * @rtpcpu: This flavor's rcu_tasks_percpu structure. +* @rtpcp_array: Array of pointers to rcu_tasks_percpu structure of CPUs in cpu_possible_mask. * @percpu_enqueue_shift: Shift down CPU ID this much when enqueuing callbacks. * @percpu_enqueue_lim: Number of per-CPU callback queues in use for enqueuing. * @percpu_dequeue_lim: Number of per-CPU callback queues in use for dequeuing.
@@ -110,6 +113,7 @@ struct rcu_tasks {call_rcu_func_t call_func;unsigned int wait_state;struct rcu_tasks_percpu __percpu *rtpcpu; +struct rcu_tasks_percpu **rtpcp_array;int percpu_enqueue_shift;int percpu_enqueue_lim;int percpu_dequeue_lim;
@@ -182,6 +186,8 @@ module_param(rcu_task_collapse_lim, int, 0444);static int rcu_task_lazy_lim__read_mostly =32 ;module_param (rcu_task_lazy_lim, int,0444 ); +static int rcu_task_cpu_ids; +/* RCU tasks grace-period state for debugging. */ #define RTGS_INIT 0#define RTGS_WAIT_WAIT_CBS 1
@@ -245,6 +251,8 @@ static void cblist_init_generic(struct rcu_tasks *rtp)int cpu;int lim;int shift; +int maxcpu; +int index =0 ;if (rcu_task_enqueue_lim <0 ) { rcu_task_enqueue_lim =1 ;
@@ -254,14 +262,9 @@ static void cblist_init_generic(struct rcu_tasks *rtp) } lim = rcu_task_enqueue_lim; -if (lim > nr_cpu_ids) - lim = nr_cpu_ids; - shift =ilog2 (nr_cpu_ids / lim); -if (((nr_cpu_ids -1 ) >> shift) >= lim) - shift++; -WRITE_ONCE (rtp->percpu_enqueue_shift, shift); -WRITE_ONCE (rtp->percpu_dequeue_lim, lim); -smp_store_release (&rtp->percpu_enqueue_lim, lim); + rtp->rtpcp_array =kcalloc (num_possible_cpus (),sizeof (struct rcu_tasks_percpu *),GFP_KERNEL ); +BUG_ON (!rtp->rtpcp_array); +for_each_possible_cpu (cpu) {struct rcu_tasks_percpu *rtpcp =per_cpu_ptr (rtp->rtpcpu, cpu);
@@ -273,14 +276,29 @@ static void cblist_init_generic(struct rcu_tasks *rtp)INIT_WORK (&rtpcp->rtp_work, rcu_tasks_invoke_cbs_wq); rtpcp->cpu = cpu; rtpcp->rtpp = rtp; + rtpcp->index = index; + rtp->rtpcp_array[index] = rtpcp; + index++;if (!rtpcp->rtp_blkd_tasks.next)INIT_LIST_HEAD (&rtpcp->rtp_blkd_tasks);if (!rtpcp->rtp_exit_list.next)INIT_LIST_HEAD (&rtpcp->rtp_exit_list); + maxcpu = cpu; } -pr_info ("%s: Setting shift to %d and lim to %d rcu_task_cb_adjust=%d.\n" , rtp->name, -data_race (rtp->percpu_enqueue_shift),data_race (rtp->percpu_enqueue_lim), rcu_task_cb_adjust); + rcu_task_cpu_ids = maxcpu +1 ; +if (lim > rcu_task_cpu_ids) + lim = rcu_task_cpu_ids; + shift =ilog2 (rcu_task_cpu_ids / lim); +if (((rcu_task_cpu_ids -1 ) >> shift) >= lim) + shift++; +WRITE_ONCE (rtp->percpu_enqueue_shift, shift); +WRITE_ONCE (rtp->percpu_dequeue_lim, lim); +smp_store_release (&rtp->percpu_enqueue_lim, lim); + +pr_info ("%s: Setting shift to %d and lim to %d rcu_task_cb_adjust=%d rcu_task_cpu_ids=%d.\n" , + rtp->name,data_race (rtp->percpu_enqueue_shift),data_race (rtp->percpu_enqueue_lim), + rcu_task_cb_adjust, rcu_task_cpu_ids); }// Compute wakeup time for lazy callback timer.
@@ -348,7 +366,7 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func, rtpcp->rtp_n_lock_retries =0 ; }if (rcu_task_cb_adjust && ++rtpcp->rtp_n_lock_retries > rcu_task_contend_lim && -READ_ONCE (rtp->percpu_enqueue_lim) != nr_cpu_ids) +READ_ONCE (rtp->percpu_enqueue_lim) != rcu_task_cpu_ids) needadjust = true;// Defer adjustment to avoid deadlock. }// Queuing callbacks before initialization not yet supported.
@@ -368,10 +386,10 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,raw_spin_unlock_irqrestore_rcu_node (rtpcp, flags);if (unlikely (needadjust)) {raw_spin_lock_irqsave (&rtp->cbs_gbl_lock, flags); -if (rtp->percpu_enqueue_lim != nr_cpu_ids) { +if (rtp->percpu_enqueue_lim != rcu_task_cpu_ids) {WRITE_ONCE (rtp->percpu_enqueue_shift,0 ); -WRITE_ONCE (rtp->percpu_dequeue_lim, nr_cpu_ids); -smp_store_release (&rtp->percpu_enqueue_lim, nr_cpu_ids); +WRITE_ONCE (rtp->percpu_dequeue_lim, rcu_task_cpu_ids); +smp_store_release (&rtp->percpu_enqueue_lim, rcu_task_cpu_ids);pr_info ("Switching %s to per-CPU callback queuing.\n" , rtp->name); }raw_spin_unlock_irqrestore (&rtp->cbs_gbl_lock, flags);
@@ -444,6 +462,8 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp) dequeue_limit =smp_load_acquire (&rtp->percpu_dequeue_lim);for (cpu =0 ; cpu < dequeue_limit; cpu++) { +if (!cpu_possible (cpu)) +continue ;struct rcu_tasks_percpu *rtpcp =per_cpu_ptr (rtp->rtpcpu, cpu);/* Advance and accelerate any new callbacks. */
@@ -481,7 +501,7 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)if (rcu_task_cb_adjust && ncbs <= rcu_task_collapse_lim) {raw_spin_lock_irqsave (&rtp->cbs_gbl_lock, flags);if (rtp->percpu_enqueue_lim >1 ) { -WRITE_ONCE (rtp->percpu_enqueue_shift,order_base_2 (nr_cpu_ids)); +WRITE_ONCE (rtp->percpu_enqueue_shift,order_base_2 (rcu_task_cpu_ids));smp_store_release (&rtp->percpu_enqueue_lim,1 ); rtp->percpu_dequeue_gpseq =get_state_synchronize_rcu (); gpdone = false;
@@ -496,7 +516,9 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)pr_info ("Completing switch %s to CPU-0 callback queuing.\n" , rtp->name); }if (rtp->percpu_dequeue_lim ==1 ) { -for (cpu = rtp->percpu_dequeue_lim; cpu < nr_cpu_ids; cpu++) { +for (cpu = rtp->percpu_dequeue_lim; cpu < rcu_task_cpu_ids; cpu++) { +if (!cpu_possible (cpu)) +continue ;struct rcu_tasks_percpu *rtpcp =per_cpu_ptr (rtp->rtpcpu, cpu);WARN_ON_ONCE (rcu_segcblist_n_cbs (&rtpcp->cblist));
@@ -511,30 +533,32 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)// Advance callbacks and invoke any that are ready. static void rcu_tasks_invoke_cbs (struct rcu_tasks *rtp,struct rcu_tasks_percpu *rtpcp) { -int cpu; -int cpunext;int cpuwq;unsigned long flags;int len; +int index;struct rcu_head *rhp;struct rcu_cblist rcl =RCU_CBLIST_INITIALIZER (rcl);struct rcu_tasks_percpu *rtpcp_next; - cpu = rtpcp->cpu; - cpunext = cpu *2 +1 ; -if (cpunext <smp_load_acquire (&rtp->percpu_dequeue_lim)) { - rtpcp_next =per_cpu_ptr (rtp->rtpcpu, cpunext); - cpuwq =rcu_cpu_beenfullyonline (cpunext) ? cpunext :WORK_CPU_UNBOUND ; -queue_work_on (cpuwq, system_wq, &rtpcp_next->rtp_work); - cpunext++; -if (cpunext <smp_load_acquire (&rtp->percpu_dequeue_lim)) { - rtpcp_next =per_cpu_ptr (rtp->rtpcpu, cpunext); - cpuwq =rcu_cpu_beenfullyonline (cpunext) ? cpunext :WORK_CPU_UNBOUND ; + index = rtpcp->index *2 +1 ; +if (index <num_possible_cpus ()) { + rtpcp_next = rtp->rtpcp_array[index]; +if (rtpcp_next->cpu <smp_load_acquire (&rtp->percpu_dequeue_lim)) { + cpuwq =rcu_cpu_beenfullyonline (rtpcp_next->cpu) ? rtpcp_next->cpu :WORK_CPU_UNBOUND ;queue_work_on (cpuwq, system_wq, &rtpcp_next->rtp_work); + index++; +if (index <num_possible_cpus ()) { + rtpcp_next = rtp->rtpcp_array[index]; +if (rtpcp_next->cpu <smp_load_acquire (&rtp->percpu_dequeue_lim)) { + cpuwq =rcu_cpu_beenfullyonline (rtpcp_next->cpu) ? rtpcp_next->cpu :WORK_CPU_UNBOUND ; +queue_work_on (cpuwq, system_wq, &rtpcp_next->rtp_work); + } + } } } -if (rcu_segcblist_empty (&rtpcp->cblist) || !cpu_possible (cpu)) +if (rcu_segcblist_empty (&rtpcp->cblist))return ;raw_spin_lock_irqsave_rcu_node (rtpcp, flags);rcu_segcblist_advance (&rtpcp->cblist,rcu_seq_current (&rtp->tasks_gp_seq));
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 7/12 rcu/tasks: Check processor-ID assumptions
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
The current mapping of smp_processor_id() to a CPU processing Tasks-RCU callbacks makes some assumptions about layout. This commit therefore adds a WARN_ON() to check these assumptions.
[ neeraj.upadhyay: Replace nr_cpu_ids with rcu_task_cpu_ids. ]
--- kernel/rcu/tasks.h | 1 + 1 file changed, 1 insertion(+)
kernel/rcu/tasks.h +1 -0
@@ -357,6 +357,7 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,rcu_read_lock (); ideal_cpu =smp_processor_id () >>READ_ONCE (rtp->percpu_enqueue_shift); chosen_cpu =cpumask_next (ideal_cpu -1 , cpu_possible_mask); +WARN_ON_ONCE (chosen_cpu >= rcu_task_cpu_ids); rtpcp =per_cpu_ptr (rtp->rtpcpu, chosen_cpu);if (!raw_spin_trylock_rcu_node (rtpcp)) {// irqs already disabled. raw_spin_lock_rcu_node (rtpcp);// irqs already disabled.
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 8/12 rcu/tasks: Update rtp->tasks_gp_seq comment
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
The rtp->tasks_gp_seq grace-period sequence number is not a strict count, but rather the usual RCU sequence number with the lower few bits tracking per-grace-period state and the upper bits the count of grace periods since boot, give or take the initial value. This commit therefore adjusts this comment.
--- kernel/rcu/tasks.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/rcu/tasks.h +1 -1
@@ -65,7 +65,7 @@ struct rcu_tasks_percpu {* @init_fract: Initial backoff sleep interval. * @gp_jiffies: Time of last @gp_state transition. * @gp_start: Most recent grace-period start in jiffies. -* @tasks_gp_seq: Number of grace periods completed since boot. +* @tasks_gp_seq: Number of grace periods completed since boot in upper bits. * @n_ipis: Number of IPIs sent to encourage grace periods to end. * @n_ipis_fails: Number of IPI-send failures. * @kthread_ptr: This flavor's grace-period/callback-invocation kthread.
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 9/12 rcu: Provide rcu_barrier_cb_is_done() to check rcu_barrier() CBs
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
This commit provides a rcu_barrier_cb_is_done() function that returns true if the *rcu_barrier*() callback passed in is done. This will be used when printing grace-period debugging information.
--- kernel/rcu/rcu.h | 5 +++++ 1 file changed, 5 insertions(+)
kernel/rcu/rcu.h +5 -0
@@ -255,6 +255,11 @@ static inline void debug_rcu_head_callback(struct rcu_head *rhp)kmem_dump_obj (rhp); } +static inline bool rcu_barrier_cb_is_done (struct rcu_head *rhp) +{ +return rhp->next == rhp; +} +extern int rcu_cpu_stall_suppress_at_boot;static inline bool rcu_stall_is_suppressed_at_boot (void )
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 10/12 rcu/tasks: Mark callbacks not currently participating in barrier operation
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
Each Tasks RCU flavor keeps a count of the number of callbacks that the current rcu_barrier_tasks*() is waiting on, but there is currently no easy way to work out which callback is stuck. One way to do this is to mark idle RCU-barrier callbacks by making the ->next pointer point to the callback itself, and this commit does just that.
Later commits will use this for debug output.
--- kernel/rcu/tasks.h | 2 ++ 1 file changed, 2 insertions(+)
kernel/rcu/tasks.h +2 -0
@@ -283,6 +283,7 @@ static void cblist_init_generic(struct rcu_tasks *rtp)INIT_LIST_HEAD (&rtpcp->rtp_blkd_tasks);if (!rtpcp->rtp_exit_list.next)INIT_LIST_HEAD (&rtpcp->rtp_exit_list); + rtpcp->barrier_q_head.next = &rtpcp->barrier_q_head; maxcpu = cpu; }
@@ -407,6 +408,7 @@ static void rcu_barrier_tasks_generic_cb(struct rcu_head *rhp)struct rcu_tasks *rtp;struct rcu_tasks_percpu *rtpcp; + rhp->next = rhp;// Mark the callback as having been invoked. rtpcp = container_of(rhp, struct rcu_tasks_percpu, barrier_q_head); rtp = rtpcp->rtpp;if (atomic_dec_and_test (&rtp->barrier_q_count))
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 11/12 rcu/tasks: Add detailed grace-period and barrier diagnostics
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
This commit adds rcu_tasks_torture_stats_print(), rcu_tasks_trace_torture_stats_print(), and rcu_tasks_rude_torture_stats_print() functions that provide detailed diagnostics on grace-period, callback, and barrier state.
--- include/linux/rcupdate.h | 3 ++ kernel/rcu/tasks.h | 66 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 67 insertions(+), 2 deletions(-)
include/linux/rcupdate.h +3 -0
@@ -164,6 +164,7 @@ static inline void rcu_nocb_flush_deferred_wakeup(void) { } } while (0) void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);void synchronize_rcu_tasks (void ); +void rcu_tasks_torture_stats_print (char *tt,char *tf);# else # define rcu_tasks_classic_qs (t, preempt) do { } while (0)# define call_rcu_tasks call_rcu
@@ -190,6 +191,7 @@ void rcu_tasks_trace_qs_blkd(struct task_struct *t); rcu_tasks_trace_qs_blkd(t); \ } \ }while (0 ) +void rcu_tasks_trace_torture_stats_print(char *tt, char *tf);# else # define rcu_tasks_trace_qs (t) do { } while (0)# endif
@@ -202,6 +204,7 @@ do { \# ifdef CONFIG_TASKS_RUDE_RCU void synchronize_rcu_tasks_rude (void ); +void rcu_tasks_rude_torture_stats_print (char *tt,char *tf);# endif #define rcu_note_voluntary_context_switch (t) rcu_tasks_qs(t, false)
kernel/rcu/tasks.h +64 -2
97 lines not on this page · raw message
Signature · 1 line
2.40.1
[email protected]
maintainerPATCH 12/12 rcu/tasks: Add rcu_barrier_tasks*() start time to diagnostics
In reply to Neeraj UpadhyayFrom: "Paul E. McKenney" <[email protected]>
This commit adds the start time, in jiffies, of the most recently started rcu_barrier_tasks*() operation to the diagnostic output used by rcuscale. This information can be helpful in distinguishing a hung barrier operation from a long series of barrier operations.
--- kernel/rcu/tasks.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
kernel/rcu/tasks.h +6 -2
@@ -87,6 +87,7 @@ struct rcu_tasks_percpu {* @barrier_q_count: Number of queues being waited on. * @barrier_q_completion: Barrier wait/wakeup mechanism. * @barrier_q_seq: Sequence number for barrier operations. +* @barrier_q_start: Most recent barrier start in jiffies. * @name: This flavor's textual name. * @kname: This flavor's kthread name. */
@@ -122,6 +123,7 @@ struct rcu_tasks {atomic_t barrier_q_count;struct completion barrier_q_completion;unsigned long barrier_q_seq; +unsigned long barrier_q_start;char *name;char *kname; };
@@ -430,6 +432,7 @@ static void __maybe_unused rcu_barrier_tasks_generic(struct rcu_tasks *rtp)mutex_unlock (&rtp->barrier_q_mutex);return ; } + rtp->barrier_q_start = jiffies;rcu_seq_start (&rtp->barrier_q_seq);init_completion (&rtp->barrier_q_completion);atomic_set (&rtp->barrier_q_count,2 );
@@ -785,8 +788,9 @@ static void rcu_tasks_torture_stats_print_generic(struct rcu_tasks *rtp, char *tpr_cont (".\n" ); else pr_cont(" (none).\n"); -pr_alert ("\tBarrier seq %lu count %d holdout CPUs " , -data_race (rtp->barrier_q_seq),atomic_read (&rtp->barrier_q_count)); +pr_alert ("\tBarrier seq %lu start %lu count %d holdout CPUs " , +data_race (rtp->barrier_q_seq), j -data_race (rtp->barrier_q_start), +atomic_read (&rtp->barrier_q_count));if (cpumask_available (cm) && !cpumask_empty (cm))pr_cont (" %*pbl.\n" ,cpumask_pr_args (cm));else
Signature · 1 line
2.40.1