rcu/nocb: De-offload and re-offload support v3
Subsystems (5)
Frederic Weisbecker
maintainercover rcu/nocb: De-offload and re-offload support v3
Hi,
This is the third attempt to make rcu-nocb mutable at runtime. See v2 there: https://lwn.net/Articles/832031/
Many fixes included and rcutorture support:
03: Whitespace fix
04: Add "rcu/nocb: Always init segcblist on CPU up" (Thanks rcutorture!)
05: Add return value for failure on (de)offloading
06: Add "rcu/nocb: Don't deoffload an offline CPU with pending work" (Thanks Paul!) 12: Add "rcu/nocb: Only cond_resched() from actual offloaded batch processing" (Thanks rcutorture!) 14: Write a real and more self-confident changelog
15/16: Add rcutorture support (thanks Paul!)
My last remaining worry, as exposed in 13/16, is that __call_rcu_core() isn't called during transition states. I need to see how I can handle that.
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
rcu/nocb-toggle-v2
HEAD: 0247cd09fd48a38741e5cc6634810985b405baca
Thanks,
Frederic
---
Frederic Weisbecker (15):
rcu: Implement rcu_segcblist_is_offloaded() config dependent
rcu: Turn enabled/offload states into a common flag
rcu: Provide basic callback offloading state machine bits rcu/nocb: Always init segcblist on CPU up
rcu: De-offloading CB kthread
rcu/nocb: Don't deoffload an offline CPU with pending work
rcu: De-offloading GP kthread
rcu: Re-offload support
rcu: Shutdown nocb timer on de-offloading
rcu: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY
rcu: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading rcu/nocb: Only cond_resched() from actual offloaded batch processing
rcu: Process batch locally as long as offloading isn't complete
rcu: Locally accelerate callbacks as long as offloading isn't complete tools/rcutorture: Support nocb toggle in TREE01
Paul E. McKenney (1):
rcutorture: Test runtime toggling of CPUs' callback offloading
Documentation/admin-guide/kernel-parameters.txt | 8 + include/linux/rcu_segcblist.h | 119 +++++++- include/linux/rcupdate.h | 4 + kernel/rcu/rcu_segcblist.c | 13 +- kernel/rcu/rcu_segcblist.h | 45 ++- kernel/rcu/rcutorture.c | 86 +++++- kernel/rcu/tree.c | 47 +-- kernel/rcu/tree.h | 2 + kernel/rcu/tree_plugin.h | 330 +++++++++++++++++++-- .../selftests/rcutorture/configs/rcu/TREE01.boot | 4 +- 10 files changed, 603 insertions(+), 55 deletions(-)
Frederic Weisbecker
maintainerPATCH 2/16 rcu: Turn enabled/offload states into a common flag
Gather the segcblist properties in a common map to avoid spreading booleans in the structure. And this prepares for the offloaded state to be mutable on runtime.
--- include/linux/rcu_segcblist.h | 6 ++++-- kernel/rcu/rcu_segcblist.c | 6 +++--- kernel/rcu/rcu_segcblist.h | 23 +++++++++++++++++++++-- 3 files changed, 28 insertions(+), 7 deletions(-)
include/linux/rcu_segcblist.h +4 -2
@@ -63,6 +63,9 @@ struct rcu_cblist {#define RCU_NEXT_TAIL 3#define RCU_CBLIST_NSEGS 4 +#define SEGCBLIST_ENABLED BIT(0) +#define SEGCBLIST_OFFLOADED BIT(1) +struct rcu_segcblist {struct rcu_head *head;struct rcu_head **tails[RCU_CBLIST_NSEGS ];
@@ -72,8 +75,7 @@ struct rcu_segcblist {#else long len;#endif -u8 enabled; -u8 offloaded; +u8 flags; };#define RCU_SEGCBLIST_INITIALIZER (n) \
kernel/rcu/rcu_segcblist.c +3 -3
@@ -152,7 +152,7 @@ void rcu_segcblist_init(struct rcu_segcblist *rsclp)for (i =0 ; i <RCU_CBLIST_NSEGS ; i++) rsclp->tails[i] = &rsclp->head;rcu_segcblist_set_len (rsclp,0 ); - rsclp->enabled =1 ; +rcu_segcblist_set_flags (rsclp,SEGCBLIST_ENABLED ); }/*
@@ -163,7 +163,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp) {WARN_ON_ONCE (!rcu_segcblist_empty (rsclp));WARN_ON_ONCE (rcu_segcblist_n_cbs (rsclp)); - rsclp->enabled =0 ; +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_ENABLED ); }/*
@@ -172,7 +172,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)*/ void rcu_segcblist_offload (struct rcu_segcblist *rsclp) { - rsclp->offloaded =1 ; +rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); }/*
kernel/rcu/rcu_segcblist.h +21 -2
@@ -50,19 +50,38 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp)#endif } +static inline void rcu_segcblist_set_flags (struct rcu_segcblist *rsclp, +int flags) +{ + rsclp->flags |= flags; +} + +static inline void rcu_segcblist_clear_flags (struct rcu_segcblist *rsclp, +int flags) +{ + rsclp->flags &= ~flags; +} + +static inline bool rcu_segcblist_test_flags (struct rcu_segcblist *rsclp, +int flags) +{ +return READ_ONCE (rsclp->flags) & flags; +} +/* * Is the specified rcu_segcblist enabled, for example, not corresponding * to an offline CPU? */ static inline bool rcu_segcblist_is_enabled (struct rcu_segcblist *rsclp) { -return rsclp->enabled; +return rcu_segcblist_test_flags (rsclp,SEGCBLIST_ENABLED ); }/* Is the specified rcu_segcblist offloaded? */ static inline bool rcu_segcblist_is_offloaded (struct rcu_segcblist *rsclp) { -return IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && rsclp->offloaded; +return IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && +rcu_segcblist_test_flags (rsclp,SEGCBLIST_OFFLOADED ); }/*
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 5/16 rcu: De-offloading CB kthread
In reply to Frederic WeisbeckerIn order to de-offload the callbacks processing of an rdp, we must clear SEGCBLIST_OFFLOAD and notify the CB kthread so that it clears its own bit flag and goes to sleep to stop handling callbacks. The GP kthread will also be notified the same way. Whoever acknowledges and clears its own bit last must notify the de-offloading worker so that it can resume the de-offloading while being sure that callbacks won't be handled remotely anymore.
--- include/linux/rcupdate.h | 2 + kernel/rcu/rcu_segcblist.c | 10 ++- kernel/rcu/rcu_segcblist.h | 2 +- kernel/rcu/tree.h | 1 + kernel/rcu/tree_plugin.h | 134 +++++++++++++++++++++++++++++++------ 5 files changed, 126 insertions(+), 23 deletions(-)
include/linux/rcupdate.h +2 -0
@@ -104,8 +104,10 @@ static inline void rcu_user_exit(void) { }#ifdef CONFIG_RCU_NOCB_CPU void rcu_init_nohz (void ); +int rcu_nocb_cpu_deoffload (int cpu);#else /* #ifdef CONFIG_RCU_NOCB_CPU */ static inline void rcu_init_nohz (void ) { } +static inline int rcu_nocb_cpu_deoffload (int cpu) {return 0 ; }#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ /**
kernel/rcu/rcu_segcblist.c +7 -3
@@ -170,10 +170,14 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)* Mark the specified rcu_segcblist structure as offloaded. This * structure must be empty. */ -void rcu_segcblist_offload (struct rcu_segcblist *rsclp) +void rcu_segcblist_offload (struct rcu_segcblist *rsclp,bool offload) { -rcu_segcblist_clear_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY ); -rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); +if (offload) { +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY ); +rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); + }else { +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_OFFLOADED ); + } }/*
kernel/rcu/rcu_segcblist.h +1 -1
@@ -105,7 +105,7 @@ static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg)void rcu_segcblist_inc_len (struct rcu_segcblist *rsclp);void rcu_segcblist_init (struct rcu_segcblist *rsclp);void rcu_segcblist_disable (struct rcu_segcblist *rsclp); -void rcu_segcblist_offload (struct rcu_segcblist *rsclp); +void rcu_segcblist_offload (struct rcu_segcblist *rsclp,bool offload);bool rcu_segcblist_ready_cbs (struct rcu_segcblist *rsclp);bool rcu_segcblist_pend_cbs (struct rcu_segcblist *rsclp);struct rcu_head *rcu_segcblist_first_cb (struct rcu_segcblist *rsclp);
kernel/rcu/tree.h +1 -0
@@ -200,6 +200,7 @@ struct rcu_data {/* 5) Callback offloading. */ #ifdef CONFIG_RCU_NOCB_CPU struct swait_queue_head nocb_cb_wq;/* For nocb kthreads to sleep on. */ +struct swait_queue_head nocb_state_wq;/* For offloading state changes */ struct task_struct *nocb_gp_kthread;raw_spinlock_t nocb_lock;/* Guard following pair of fields. */ atomic_t nocb_lock_contended;/* Contention experienced. */
kernel/rcu/tree_plugin.h +115 -19
182 lines not on this page · raw message
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 7/16 rcu: De-offloading GP kthread
In reply to Frederic WeisbeckerIn order to de-offload the callbacks processing of an rdp, we must clear SEGCBLIST_OFFLOAD and notify the GP kthread so that it clears its own bit flag and ignore the target rdp from its loop. The CB kthread is also notified the same way. Whoever acknowledges and clears its own bit last must notify the de-offloading worker so that it can resume the de-offloading while being sure that callbacks won't be handled remotely anymore.
--- kernel/rcu/tree_plugin.h | 54 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 51 insertions(+), 3 deletions(-)
kernel/rcu/tree_plugin.h +51 -3
@@ -1928,6 +1928,33 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)__call_rcu_nocb_wake (rdp, true, flags); } +static inline bool nocb_gp_enabled_cb (struct rcu_data *rdp) +{ +u8 flags =SEGCBLIST_OFFLOADED |SEGCBLIST_KTHREAD_GP ; + +return rcu_segcblist_test_flags (&rdp->cblist, flags); +} + +static inline bool nocb_gp_update_state (struct rcu_data *rdp,bool *needwake_state) +{ +struct rcu_segcblist *cblist = &rdp->cblist; + +if (rcu_segcblist_test_flags (cblist,SEGCBLIST_OFFLOADED )) { +return true; + }else { +/* +* De-offloading. Clear our flag and notify the de-offload worker. +* We will ignore this rdp until it ever gets re-offloaded. +*/ +WARN_ON_ONCE (!rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_GP )); +rcu_segcblist_clear_flags (cblist,SEGCBLIST_KTHREAD_GP ); +if (!rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB )) + *needwake_state = true; +return false; + } +} + +/* * No-CBs GP kthreads come here to wait for additional callbacks to show up * or for grace periods to end.
@@ -1956,8 +1983,17 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)*/ WARN_ON_ONCE (my_rdp->nocb_gp_rdp != my_rdp);for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) { +bool needwake_state = false; +if (!nocb_gp_enabled_cb (rdp)) +continue ;trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("Check" ));rcu_nocb_lock_irqsave (rdp, flags); +if (!nocb_gp_update_state (rdp, &needwake_state)) { +rcu_nocb_unlock_irqrestore (rdp, flags); +if (needwake_state) +swake_up_one (&rdp->nocb_state_wq); +continue ; + } bypass_ncbs =rcu_cblist_n_cbs (&rdp->nocb_bypass); if (bypass_ncbs && (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -2221,8 +2257,9 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)static int __rcu_nocb_rdp_deoffload (struct rcu_data *rdp) {struct rcu_segcblist *cblist = &rdp->cblist; +struct rcu_data *rdp_gp = rdp->nocb_gp_rdp; +bool wake_cb = false, wake_gp = false;struct rcu_node *rnp = rdp->mynode; -bool wake_cb = false;unsigned long flags;printk ("De-offloading %d\n" , rdp->cpu);
@@ -2249,9 +2286,19 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)if (wake_cb)swake_up_one (&rdp->nocb_cb_wq); +raw_spin_lock_irqsave (&rdp_gp->nocb_gp_lock, flags); +if (rdp_gp->nocb_gp_sleep) { + rdp_gp->nocb_gp_sleep = false; + wake_gp = true; + } +raw_spin_unlock_irqrestore (&rdp_gp->nocb_gp_lock, flags); + +if (wake_gp) +wake_up_process (rdp_gp->nocb_gp_kthread); +swait_event_exclusive (rdp->nocb_state_wq, - !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB )); - + !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB | +SEGCBLIST_KTHREAD_GP ));return 0 ; }
@@ -2333,6 +2380,7 @@ void __init rcu_init_nohz(void)rcu_segcblist_init (&rdp->cblist);rcu_segcblist_offload (&rdp->cblist, true);rcu_segcblist_set_flags (&rdp->cblist,SEGCBLIST_KTHREAD_CB ); +rcu_segcblist_set_flags (&rdp->cblist,SEGCBLIST_KTHREAD_GP ); }rcu_organize_nocb_kthreads (); }
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 10/16 rcu: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY
In reply to Frederic WeisbeckerMake sure to handle the pending bypass queue before we switch to the final de-offload state. We'll have to be careful and later set SEGCBLIST_SOFTIRQ_ONLY before re-enabling again IRQs, or new bypass callbacks could be queued in the meantine.
--- kernel/rcu/tree_plugin.h | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/rcu/tree_plugin.h +9 -2
@@ -2342,12 +2342,19 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)swait_event_exclusive (rdp->nocb_state_wq, !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP )); +rcu_nocb_lock_irqsave (rdp, flags);/* Make sure nocb timer won't stay around */ -rcu_nocb_lock_irqsave (rdp, flags);WRITE_ONCE (rdp->nocb_defer_wakeup,RCU_NOCB_WAKE_OFF ); -rcu_nocb_unlock_irqrestore (rdp, flags);del_timer_sync (&rdp->nocb_timer); +/* +* Flush bypass. While IRQs are disabled and once we set +* SEGCBLIST_SOFTIRQ_ONLY, no callback is supposed to be +* enqueued on bypass. +*/ +rcu_nocb_flush_bypass (rdp,NULL , jiffies); +rcu_nocb_unlock_irqrestore (rdp, flags); +return ret; }
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 3/16 rcu: Provide basic callback offloading state machine bits
In reply to Frederic WeisbeckerWe'll need to be able to runtime offload and de-offload the processing of callback for a given CPU. In order to support a smooth transition from unlocked local processing (softirq/rcuc) to locked offloaded processing (rcuop/rcuog) and the reverse, provide the necessary bits and documentation for the state machine that will carry up all the steps to enforce correctness while serving callbacks processing all along.
--- include/linux/rcu_segcblist.h | 115 +++++++++++++++++++++++++++++++++- kernel/rcu/rcu_segcblist.c | 1 + kernel/rcu/rcu_segcblist.h | 12 +++- kernel/rcu/tree.c | 3 + 4 files changed, 128 insertions(+), 3 deletions(-)
include/linux/rcu_segcblist.h +114 -1
@@ -63,8 +63,121 @@ struct rcu_cblist {#define RCU_NEXT_TAIL 3#define RCU_CBLIST_NSEGS 4 + +/* +* ==NOCB Offloading state machine== +* +* +* ---------------------------------------------------------------------------- +* | SEGCBLIST_SOFTIRQ_ONLY | +* | | +* | Callbacks processed by rcu_core() from softirqs or local | +* | rcuc kthread, without holding nocb_lock. | +* ---------------------------------------------------------------------------- +* | +* v +* ---------------------------------------------------------------------------- +* | SEGCBLIST_OFFLOADED | +* | | +* | Callbacks processed by rcu_core() from softirqs or local | +* | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads, | +* | allowing nocb_timer to be armed. | +* ---------------------------------------------------------------------------- +* | +* v +* ----------------------------------- +* | | +* v v +* --------------------------------------- ----------------------------------| +* | SEGCBLIST_OFFLOADED | | | SEGCBLIST_OFFLOADED | | +* | SEGCBLIST_KTHREAD_CB | | SEGCBLIST_KTHREAD_GP | +* | | | | +* | | | | +* | CB kthread woke up and | | GP kthread woke up and | +* | acknowledged SEGCBLIST_OFFLOADED. | | acknowledged SEGCBLIST_OFFLOADED| +* | Processes callbacks concurrently | | | +* | with rcu_core(), holding | | | +* | nocb_lock. | | | +* --------------------------------------- ----------------------------------- +* | | +* ----------------------------------- +* | +* v +* |--------------------------------------------------------------------------| +* | SEGCBLIST_OFFLOADED | | +* | SEGCBLIST_KTHREAD_CB | | +* | SEGCBLIST_KTHREAD_GP | +* | | +* | Kthreads handle callbacks holding nocb_lock, local rcu_core() stops | +* | handling callbacks. | +* ---------------------------------------------------------------------------- +*/ + + + +/* +* ==NOCB De-Offloading state machine== +* +* +* |--------------------------------------------------------------------------| +* | SEGCBLIST_OFFLOADED | | +* | SEGCBLIST_KTHREAD_CB | | +* | SEGCBLIST_KTHREAD_GP | +* | | +* | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() | +* | ignores callbacks. | +* ---------------------------------------------------------------------------- +* | +* v +* |--------------------------------------------------------------------------| +* | SEGCBLIST_KTHREAD_CB | | +* | SEGCBLIST_KTHREAD_GP | +* | | +* | CB/GP kthreads and local rcu_core() handle callbacks concurrently | +* | holding nocb_lock. Wake up CB and GP kthreads if necessary. | +* ---------------------------------------------------------------------------- +* | +* v +* ----------------------------------- +* | | +* v v +* ---------------------------------------------------------------------------| +* | | +* | SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP | +* | | | +* | GP kthread woke up and | CB kthread woke up and | +* | acknowledged the fact that | acknowledged the fact that | +* | SEGCBLIST_OFFLOADED got cleared. | SEGCBLIST_OFFLOADED got cleared. | +* | | The CB kthread goes to sleep | +* | The callbacks from the target CPU | until it ever gets re-offloaded. | +* | will be ignored from the GP kthread | | +* | loop. | | +* ---------------------------------------------------------------------------- +* | | +* ----------------------------------- +* | +* v +* ---------------------------------------------------------------------------- +* | 0 | +* | | +* | Callbacks processed by rcu_core() from softirqs or local | +* | rcuc kthread, while holding nocb_lock. Forbid nocb_timer to be armed. | +* | Flush pending nocb_timer. Flush nocb bypass callbacks. | +* ---------------------------------------------------------------------------- +* | +* v +* ---------------------------------------------------------------------------- +* | SEGCBLIST_SOFTIRQ_ONLY | +* | | +* | Callbacks processed by rcu_core() from softirqs or local | +* | rcuc kthread, without holding nocb_lock. | +* ---------------------------------------------------------------------------- +*/ #define SEGCBLIST_ENABLED BIT(0) -#define SEGCBLIST_OFFLOADED BIT(1) +#define SEGCBLIST_SOFTIRQ_ONLY BIT(1) +#define SEGCBLIST_KTHREAD_CB BIT(2) +#define SEGCBLIST_KTHREAD_GP BIT(3) +#define SEGCBLIST_OFFLOADED BIT(4)struct rcu_segcblist {struct rcu_head *head;
kernel/rcu/rcu_segcblist.c +1 -0
@@ -172,6 +172,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)*/ void rcu_segcblist_offload (struct rcu_segcblist *rsclp) { +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY );rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); }
kernel/rcu/rcu_segcblist.h +10 -2
@@ -80,8 +80,16 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)/* Is the specified rcu_segcblist offloaded? */ static inline bool rcu_segcblist_is_offloaded (struct rcu_segcblist *rsclp) { -return IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_test_flags (rsclp,SEGCBLIST_OFFLOADED ); +if (IS_ENABLED (CONFIG_RCU_NOCB_CPU )) { +/* +* Complete de-offloading happens only when SEGCBLIST_SOFTIRQ_ONLY +* is set. +*/ +if (!rcu_segcblist_test_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY )) +return true; + } + +return false; }/*
kernel/rcu/tree.c +3 -0
@@ -83,6 +83,9 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = { .dynticks_nesting = 1, .dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE, .dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR), +#ifdef CONFIG_RCU_NOCB_CPU + .cblist.flags = SEGCBLIST_SOFTIRQ_ONLY, +#endif };static struct rcu_state rcu_state = { .level = { &rcu_state.node[0 ] },
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 9/16 rcu: Shutdown nocb timer on de-offloading
In reply to Frederic WeisbeckerMake sure the nocb timer can't fire anymore before we reach the final de-offload state. Spuriously waking up the GP kthread is no big deal but we must prevent from executing the timer callback without nocb locking.
--- kernel/rcu/tree.h | 1 + kernel/rcu/tree_plugin.h | 12 +++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-)
kernel/rcu/tree.h +1 -0
@@ -256,6 +256,7 @@ struct rcu_data { };/* Values for nocb_defer_wakeup field in struct rcu_data. */ +#define RCU_NOCB_WAKE_OFF -1#define RCU_NOCB_WAKE_NOT 0#define RCU_NOCB_WAKE 1#define RCU_NOCB_WAKE_FORCE 2
kernel/rcu/tree_plugin.h +11 -1
@@ -1665,6 +1665,8 @@ static void wake_nocb_gp(struct rcu_data *rdp, bool force,static void wake_nocb_gp_defer (struct rcu_data *rdp,int waketype,const char *reason) { +if (rdp->nocb_defer_wakeup ==RCU_NOCB_WAKE_OFF ) +return ;if (rdp->nocb_defer_wakeup ==RCU_NOCB_WAKE_NOT )mod_timer (&rdp->nocb_timer, jiffies +1 );if (rdp->nocb_defer_wakeup < waketype)
@@ -2243,7 +2245,7 @@ static int rcu_nocb_cb_kthread(void *arg)/* Is a deferred wakeup of rcu_nocb_kthread() required? */ static int rcu_nocb_need_deferred_wakeup (struct rcu_data *rdp) { -return READ_ONCE (rdp->nocb_defer_wakeup); +return READ_ONCE (rdp->nocb_defer_wakeup) >RCU_NOCB_WAKE_NOT ; }/* Do a deferred wakeup of rcu_nocb_kthread(). */
@@ -2340,6 +2342,12 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)swait_event_exclusive (rdp->nocb_state_wq, !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP )); +/* Make sure nocb timer won't stay around */ +rcu_nocb_lock_irqsave (rdp, flags); +WRITE_ONCE (rdp->nocb_defer_wakeup,RCU_NOCB_WAKE_OFF ); +rcu_nocb_unlock_irqrestore (rdp, flags); +del_timer_sync (&rdp->nocb_timer); +return ret; }
@@ -2396,6 +2404,8 @@ static int __rcu_nocb_rdp_offload(struct rcu_data *rdp)* SEGCBLIST_SOFTIRQ_ONLY mode. */ raw_spin_lock_irqsave (&rdp->nocb_lock, flags); +/* Re-enable nocb timer */ +WRITE_ONCE (rdp->nocb_defer_wakeup,RCU_NOCB_WAKE_NOT );/* * We didn't take the nocb lock while working on the * rdp->cblist in SEGCBLIST_SOFTIRQ_ONLY mode.
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 8/16 rcu: Re-offload support
In reply to Frederic WeisbeckerIn order to re-offload the callbacks processing of an rdp, we must clear SEGCBLIST_SOFTIRQ_ONLY, set SEGCBLIST_OFFLOADED and notify the CB and GP kthreads so that they both set their own bit flag and start processing the callbacks remotely. The re-offloading worker is then notified that it can stop processing the callbacks locally.
Ordering must be carefully enforced so that the callbacks that used to be processed locally without locking must have their latest updates visible by the time they get processed by the kthreads.
--- include/linux/rcupdate.h | 2 + kernel/rcu/tree_plugin.h | 157 +++++++++++++++++++++++++++++++++------ 2 files changed, 138 insertions(+), 21 deletions(-)
include/linux/rcupdate.h +2 -0
@@ -104,9 +104,11 @@ static inline void rcu_user_exit(void) { }#ifdef CONFIG_RCU_NOCB_CPU void rcu_init_nohz (void ); +int rcu_nocb_cpu_offload (int cpu);int rcu_nocb_cpu_deoffload (int cpu);#else /* #ifdef CONFIG_RCU_NOCB_CPU */ static inline void rcu_init_nohz (void ) { } +static inline int rcu_nocb_cpu_offload (int cpu) {return -EINVAL ; }static inline int rcu_nocb_cpu_deoffload (int cpu) {return 0 ; }#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
kernel/rcu/tree_plugin.h +136 -21
219 lines not on this page · raw message
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 11/16 rcu: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading
In reply to Frederic WeisbeckerSet SEGCBLIST_SOFTIRQ_ONLY once everything is settled. After that, the callbacks are handled locklessly and locally.
--- kernel/rcu/tree_plugin.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +8 -1
@@ -2353,7 +2353,14 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)* enqueued on bypass. */ rcu_nocb_flush_bypass (rdp,NULL , jiffies); -rcu_nocb_unlock_irqrestore (rdp, flags); +rcu_segcblist_set_flags (cblist,SEGCBLIST_SOFTIRQ_ONLY ); +/* +* With SEGCBLIST_SOFTIRQ_ONLY, we can't use +* rcu_nocb_unlock_irqrestore() anymore. Theoretically we +* could set SEGCBLIST_SOFTIRQ_ONLY with cb unlocked and IRQs +* disabled now, but let's be paranoid. +*/ +raw_spin_unlock_irqrestore (&rdp->nocb_lock, flags);return ret; }
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 12/16 rcu/nocb: Only cond_resched() from actual offloaded batch processing
In reply to Frederic Weisbeckerrcu_do_batch() will be callable concurrently by softirqs and offloaded processing. So make sure we actually call cond resched only from the offloaded context.
--- kernel/rcu/tree.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/rcu/tree.c +1 -2
@@ -2491,8 +2491,7 @@ static void rcu_do_batch(struct rcu_data *rdp)/* Exceeded the time limit, so leave. */ break ; } -if (offloaded) { -WARN_ON_ONCE (in_serving_softirq ()); +if (!in_serving_softirq ()) {local_bh_enable ();lockdep_assert_irqs_enabled ();cond_resched_tasks_rcu_qs ();
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 6/16 rcu/nocb: Don't deoffload an offline CPU with pending work
In reply to Frederic WeisbeckerOfflining offloaded CPUs don't migrate their callbacks just like non-offloaded CPUs do. It's up to their CB/GP kthread to handle what remains.
Therefore we can't afford to de-offload an offline CPU that still has pending work to do, or the callbacks would be ignored.
NOTE: The long term solution will be to wait for all pending callbacks to be processed before completing a CPU down operation.
--- kernel/rcu/tree_plugin.h | 8 ++++++++ 1 file changed, 8 insertions(+)
kernel/rcu/tree_plugin.h +8 -0
@@ -2228,6 +2228,14 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)printk ("De-offloading %d\n" , rdp->cpu);rcu_nocb_lock_irqsave (rdp, flags); +/* +* If there are still pending work offloaded, the offline +* CPU won't help much handling them. +*/ +if (cpu_is_offline (rdp->cpu) && !rcu_segcblist_empty (&rdp->cblist)) { +rcu_nocb_unlock_irqrestore (rdp, flags); +return -EBUSY ; + }raw_spin_lock_rcu_node (rnp);rcu_segcblist_offload (cblist, false);raw_spin_unlock_rcu_node (rnp);
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 13/16 rcu: Process batch locally as long as offloading isn't complete
In reply to Frederic WeisbeckerDuring the offloading or de-offloading process, make sure to process the callbacks batch locally whenever the segcblist isn't entirely offloaded. This enforces callback service processing while we are still in intermediate (de-)offloading state.
FIXME: Note that __call_rcu_core() isn't called during these intermediate states. Some pieces there may still be necessary.
--- kernel/rcu/rcu_segcblist.h | 12 ++++++++++++ kernel/rcu/tree.c | 3 ++- 2 files changed, 14 insertions(+), 1 deletion(-)
kernel/rcu/rcu_segcblist.h +12 -0
@@ -92,6 +92,18 @@ static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)return false; } +static inline bool rcu_segcblist_completely_offloaded (struct rcu_segcblist *rsclp) +{ +int flags =SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP |SEGCBLIST_OFFLOADED ; + +if (IS_ENABLED (CONFIG_RCU_NOCB_CPU )) { +if ((rsclp->flags & flags) == flags) +return true; + } + +return false; +} +/* * Are all segments following the specified segment of the specified * rcu_segcblist structure empty of callbacks? (The specified
kernel/rcu/tree.c +2 -1
@@ -2674,6 +2674,7 @@ static __latent_entropy void rcu_core(void)struct rcu_data *rdp =raw_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode;const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist); +const bool do_batch = !rcu_segcblist_completely_offloaded (&rdp->cblist);if (cpu_is_offline (smp_processor_id ()))return ;
@@ -2703,7 +2704,7 @@ static __latent_entropy void rcu_core(void)rcu_check_gp_start_stall (rnp, rdp,rcu_jiffies_till_stall_check ());/* If there are callbacks ready, invoke them. */ -if (!offloaded &&rcu_segcblist_ready_cbs (&rdp->cblist) && +if (do_batch &&rcu_segcblist_ready_cbs (&rdp->cblist) &&likely (READ_ONCE (rcu_scheduler_fully_active)))rcu_do_batch (rdp);
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 14/16 rcu: Locally accelerate callbacks as long as offloading isn't complete
In reply to Frederic WeisbeckerThe local callbacks processing checks if some callbacks need acceleration. Keep that behaviour under nocb lock protection when rcu_core() executes concurrently with GP/CB kthreads.
--- kernel/rcu/tree.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +3 -4
@@ -2673,7 +2673,6 @@ static __latent_entropy void rcu_core(void)unsigned long flags;struct rcu_data *rdp =raw_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; -const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);const bool do_batch = !rcu_segcblist_completely_offloaded (&rdp->cblist);if (cpu_is_offline (smp_processor_id ()))
@@ -2694,11 +2693,11 @@ static __latent_entropy void rcu_core(void)/* No grace period and unregistered callbacks? */ if (!rcu_gp_in_progress () && -rcu_segcblist_is_enabled (&rdp->cblist) && !offloaded) { -local_irq_save (flags); +rcu_segcblist_is_enabled (&rdp->cblist) && do_batch) { +rcu_nocb_lock_irqsave (rdp, flags);if (!rcu_segcblist_restempty (&rdp->cblist,RCU_NEXT_READY_TAIL ))rcu_accelerate_cbs_unlocked (rnp, rdp); -local_irq_restore (flags); +rcu_nocb_unlock_irqrestore (rdp, flags); }rcu_check_gp_start_stall (rnp, rdp,rcu_jiffies_till_stall_check ());
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 16/16 tools/rcutorture: Support nocb toggle in TREE01
In reply to Frederic WeisbeckerAdd periodic toggling of 7 CPUs over 8 every second in order to test NOCB toggle code. Choose TREE01 for that as it's already testing nocb.
--- tools/testing/selftests/rcutorture/configs/rcu/TREE01.boot | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/rcutorture/configs/rcu/TREE01.boot +3 -1
@@ -2,5 +2,7 @@ maxcpus=8 nr_cpus=43 rcutree.gp_preinit_delay=3 rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 -rcu_nocbs=0 +rcu_nocbs=0-1,3-7 +rcutorture.nocbs_nthreads=8 +rcutorture.nocbs_toggle=1000 rcutorture.fwd_progress=0
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 15/16 rcutorture: Test runtime toggling of CPUs' callback offloading
In reply to Frederic WeisbeckerFrom: "Paul E. McKenney" <[email protected]>
Frederic Weisbecker is adding the ability to change the rcu_nocbs state of CPUs at runtime, that is, to offload and deoffload their RCU callback processing without the need to reboot. As the old saying goes, "if it ain't tested, it don't work", so this commit therefore adds prototype rcutorture testing for this capability.
--- .../admin-guide/kernel-parameters.txt | 8 ++ kernel/rcu/rcutorture.c | 86 ++++++++++++++++++- kernel/rcu/tree_plugin.h | 3 +- 3 files changed, 93 insertions(+), 4 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -4324,6 +4324,14 @@ stress RCU, they don't participate in the actual test, hence the "fake". + rcutorture.nocbs_nthreads= [KNL] + Set number of RCU callback-offload togglers. + Zero (the default) disables toggling. + + rcutorture.nocbs_toggle= [KNL] + Set the delay in milliseconds between successive + callback-offload toggling attempts. + rcutorture.nreaders= [KNL] Set number of RCU readers. The value -1 selects N-1, where N is the number of CPUs. A value
kernel/rcu/rcutorture.c +83 -3
144 lines not on this page · raw message
kernel/rcu/tree_plugin.h +2 -1
@@ -2398,6 +2398,7 @@ int rcu_nocb_cpu_deoffload(int cpu)return ret; } +EXPORT_SYMBOL_GPL (rcu_nocb_cpu_deoffload);static int __rcu_nocb_rdp_offload (struct rcu_data *rdp) {
@@ -2473,7 +2474,7 @@ int rcu_nocb_cpu_offload(int cpu)return ret; } - +EXPORT_SYMBOL_GPL (rcu_nocb_cpu_offload);void __init rcu_init_nohz (void ) {
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 4/16 rcu/nocb: Always init segcblist on CPU up
In reply to Frederic WeisbeckerAn rdp's segcblist enabled state is treated differently on CPU hotplug operations, depending on whether it is offloaded or not.
1) Not offloaded: An rdp is disabled on CPU down. All its callbacks are migrated and no more aren't supposed to be enqueued until it gets re-enabled on CPU up.
2) Offloaded: An rdp is not disabled on CPU down in order to let the CB/GP kthreads finish their jobs on remaining callbacks. Hence it is not re-enabled on CPU up either.
Since an rdp's offloaded state is set in stone at boot, we expect the offloaded state to remain the same between CPU down and CPU up. So 1) and 2) are symmetrical.
Now the offloaded state will become toggable at runtime. Hence the new possible asymmetrical scenarios:
3) An rdp goes into CPU down while in a not-offloaded state. It gets later set to offloaded and finally goes into CPU up.
4) An rdp goes into CPU down while in an offloaded state. It gets later set to not-offloaded and finally goes into CPU up.
The scenario 4) is currently well handled. The rdp isn't disabled on CPU down and it gets re-initialized on CPU up. We require the segcblist to be empty in order to toggle to non-offloaded state while a CPU is offlined.
The scenario 3) would run into trouble though, as the rdp is disabled on CPU down and not re-initialized/re-enabled on CPU up.
In order to fix this, always re-initialize/re-enable an rdp on CPU up unless it still has callbacks at that time, which anyway can only happen when the rdp went down and up in offloaded state (case 2), the only case that doesn't need re-initialization.
NOTE: The proper longer term fix will be to wait for all the offloaded callbacks to be processed before completing CPU down operations. So we can unconditionally re-initialize on CPU up.
--- kernel/rcu/tree.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/rcu/tree.c +9 -3
@@ -3940,12 +3940,18 @@ int rcutree_prepare_cpu(unsigned int cpu) rdp->qlen_last_fqs_check =0 ; rdp->n_force_qs_snap = rcu_state.n_force_qs; rdp->blimit = blimit; -if (rcu_segcblist_empty (&rdp->cblist) &&/* No early-boot CBs? */ - !rcu_segcblist_is_offloaded (&rdp->cblist)) -rcu_segcblist_init (&rdp->cblist);/* Re-enable callbacks. */ rdp->dynticks_nesting =1 ;/* CPU not up, no tearing. */ rcu_dynticks_eqs_online ();raw_spin_unlock_rcu_node (rnp);/* irqs remain disabled. */ +/* +* Lock in case the CB/GP kthreads are still around handling +* old callbacks (longer term we should flush all callbacks +* before completing CPU offline) +*/ +rcu_nocb_lock (rdp); +if (rcu_segcblist_empty (&rdp->cblist))/* No early-boot CBs? */ +rcu_segcblist_init (&rdp->cblist);/* Re-enable callbacks. */ +rcu_nocb_unlock (rdp);/* * Add CPU to leaf rcu_node pending-online bitmask. Any needed
Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerPATCH 1/16 rcu: Implement rcu_segcblist_is_offloaded() config dependent
In reply to Frederic WeisbeckerThis simplify the usage of this API and avoid checking the kernel config from the callers.
--- kernel/rcu/rcu_segcblist.h | 2 +- kernel/rcu/tree.c | 21 +++++++-------------- 2 files changed, 8 insertions(+), 15 deletions(-)
kernel/rcu/rcu_segcblist.h +1 -1
@@ -62,7 +62,7 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)/* Is the specified rcu_segcblist offloaded? */ static inline bool rcu_segcblist_is_offloaded (struct rcu_segcblist *rsclp) { -return rsclp->offloaded; +return IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && rsclp->offloaded; }/*
kernel/rcu/tree.c +7 -14
@@ -1603,8 +1603,7 @@ static bool __note_gp_changes(struct rcu_node *rnp, struct rcu_data *rdp) {bool ret = false;bool need_qs; -const bool offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); +const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);raw_lockdep_assert_held_rcu_node (rnp);
@@ -2048,8 +2047,7 @@ static void rcu_gp_cleanup(void) needgp = true; }/* Advance CBs to reduce false positives below. */ - offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); + offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);if ((offloaded || !rcu_accelerate_cbs (rnp, rdp)) && needgp) {WRITE_ONCE (rcu_state.gp_flags,RCU_GP_FLAG_INIT );WRITE_ONCE (rcu_state.gp_req_activity, jiffies);
@@ -2248,8 +2246,7 @@ rcu_report_qs_rdp(struct rcu_data *rdp)unsigned long flags;unsigned long mask;bool needwake = false; -const bool offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); +const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);struct rcu_node *rnp;WARN_ON_ONCE (rdp->cpu !=smp_processor_id ());
@@ -2417,8 +2414,7 @@ static void rcu_do_batch(struct rcu_data *rdp) {int div;unsigned long flags; -const bool offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); +const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);struct rcu_head *rhp;struct rcu_cblist rcl =RCU_CBLIST_INITIALIZER (rcl);long bl, count;
@@ -2675,8 +2671,7 @@ static __latent_entropy void rcu_core(void)unsigned long flags;struct rcu_data *rdp =raw_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; -const bool offloaded =IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_is_offloaded (&rdp->cblist); +const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);if (cpu_is_offline (smp_processor_id ()))return ;
@@ -2978,8 +2973,7 @@ __call_rcu(struct rcu_head *head, rcu_callback_t func) rcu_segcblist_n_cbs(&rdp->cblist));/* Go handle any RCU core processing required. */ -if (IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -unlikely (rcu_segcblist_is_offloaded (&rdp->cblist))) { +if (unlikely (rcu_segcblist_is_offloaded (&rdp->cblist))) {__call_rcu_nocb_wake (rdp, was_alldone, flags);/* unlocks */ }else {__call_rcu_core (rdp, head, flags);
@@ -3712,8 +3706,7 @@ static int rcu_pending(int user)/* Has RCU gone idle with this CPU needing another grace period? */ if (!gp_in_progress &&rcu_segcblist_is_enabled (&rdp->cblist) && - (!IS_ENABLED (CONFIG_RCU_NOCB_CPU ) || - !rcu_segcblist_is_offloaded (&rdp->cblist)) && + !rcu_segcblist_is_offloaded (&rdp->cblist) && !rcu_segcblist_restempty (&rdp->cblist,RCU_NEXT_READY_TAIL ))return 1 ;
Signature · 1 line
2.25.1
Re: [PATCH 05/16] rcu: De-offloading CB kthread
In reply to Frederic WeisbeckerHi Frederic,
Could you copy the [email protected] if you have another version, it will help RCU hobbyists like me to catch up news in RCU, thanks! ;-)
Please see below for some comments, I'm still reading the whole patchset, so probably I miss something..
Quoted 195 lines · Frederic Weisbecker
In order to de-offload the callbacks processing of an rdp, we must clear
SEGCBLIST_OFFLOADand notify the CB kthread so that it clears its own bit flag and goes to sleep to stop handling callbacks. The GP kthread will also be notified the same way. Whoever acknowledges and clears its own bit last must notify the de-offloading worker so that it can resume the de-offloading while being sure that callbacks won't be handled remotely anymore.Inspired-by Paul E. McKenney <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>Cc Paul E. McKenney <[email protected]>Cc Josh Triplett <[email protected]>Cc Steven Rostedt <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Joel Fernandes <[email protected]>Cc Neeraj Upadhyay <[email protected]>--- include/linux/rcupdate.h | 2 + kernel/rcu/rcu_segcblist.c | 10 ++- kernel/rcu/rcu_segcblist.h | 2 +- kernel/rcu/tree.h | 1 + kernel/rcu/tree_plugin.h | 134 +++++++++++++++++++++++++++++++------ 5 files changed, 126 insertions(+), 23 deletions(-)include/linux/rcupdate.h +2
-0@@ -104,8 +104,10 @@ static inline void rcu_user_exit(void) { }#ifdef CONFIG_RCU_NOCB_CPU void rcu_init_nohz (void ); +int rcu_nocb_cpu_deoffload (int cpu);#else /* #ifdef CONFIG_RCU_NOCB_CPU */ static inline void rcu_init_nohz (void ) { } +static inline int rcu_nocb_cpu_deoffload (int cpu) {return 0 ; }#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ /** kernel/rcu/rcu_segcblist.c +7
-3@@ -170,10 +170,14 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)* Mark the specified rcu_segcblist structure as offloaded. This * structure must be empty. */ -void rcu_segcblist_offload (struct rcu_segcblist *rsclp) +void rcu_segcblist_offload (struct rcu_segcblist *rsclp,bool offload) { -rcu_segcblist_clear_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY ); -rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); +if (offload) { +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY ); +rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); + }else { +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_OFFLOADED ); + } }/* kernel/rcu/rcu_segcblist.h +1
-1@@ -105,7 +105,7 @@ static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg)void rcu_segcblist_inc_len (struct rcu_segcblist *rsclp);void rcu_segcblist_init (struct rcu_segcblist *rsclp);void rcu_segcblist_disable (struct rcu_segcblist *rsclp); -void rcu_segcblist_offload (struct rcu_segcblist *rsclp); +void rcu_segcblist_offload (struct rcu_segcblist *rsclp,bool offload);bool rcu_segcblist_ready_cbs (struct rcu_segcblist *rsclp);bool rcu_segcblist_pend_cbs (struct rcu_segcblist *rsclp);struct rcu_head *rcu_segcblist_first_cb (struct rcu_segcblist *rsclp);kernel/rcu/tree.h +1
-0@@ -200,6 +200,7 @@ struct rcu_data {/* 5) Callback offloading. */ #ifdef CONFIG_RCU_NOCB_CPU struct swait_queue_head nocb_cb_wq;/* For nocb kthreads to sleep on. */ +struct swait_queue_head nocb_state_wq;/* For offloading state changes */ struct task_struct *nocb_gp_kthread;raw_spinlock_t nocb_lock;/* Guard following pair of fields. */ atomic_t nocb_lock_contended;/* Contention experienced. */ kernel/rcu/tree_plugin.h +50
-18@@ -2081,16 +2081,29 @@ static int rcu_nocb_gp_kthread(void *arg)return 0 ; } +static inline bool nocb_cb_can_run (struct rcu_data *rdp) +{ +u8 flags =SEGCBLIST_OFFLOADED |SEGCBLIST_KTHREAD_CB ; +return rcu_segcblist_test_flags (&rdp->cblist, flags); +} + +static inline bool nocb_cb_wait_cond (struct rcu_data *rdp) +{ +return nocb_cb_can_run (rdp) && !READ_ONCE (rdp->nocb_cb_sleep); +} +/* * Invoke any ready callbacks from the corresponding no-CBs CPU, * then, if there are no more, wait for more to appear. */ static void nocb_cb_wait (struct rcu_data *rdp) { -unsigned long cur_gp_seq; -unsigned long flags; +struct rcu_segcblist *cblist = &rdp->cblist; +struct rcu_node *rnp = rdp->mynode; +bool needwake_state = false;bool needwake_gp = false; -struct rcu_node *rnp = rdp->mynode; +unsigned long cur_gp_seq; +unsigned long flags;local_irq_save (flags);rcu_momentary_dyntick_idle ();@@ -2100,32 +2113,50 @@ static void nocb_cb_wait(struct rcu_data *rdp)local_bh_enable ();lockdep_assert_irqs_enabled ();rcu_nocb_lock_irqsave (rdp, flags); -if (rcu_segcblist_nextgp (&rdp->cblist, &cur_gp_seq) && +if (rcu_segcblist_nextgp (cblist, &cur_gp_seq) &&rcu_seq_done (&rnp->gp_seq, cur_gp_seq) &&raw_spin_trylock_rcu_node (rnp)) {/* irqs already disabled. */ needwake_gp =rcu_advance_cbs (rdp->mynode, rdp);raw_spin_unlock_rcu_node (rnp);/* irqs remain disabled. */ } -if (rcu_segcblist_ready_cbs (&rdp->cblist)) { -rcu_nocb_unlock_irqrestore (rdp, flags); -if (needwake_gp) -rcu_gp_kthread_wake (); -return ; - } -trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("CBSleep" ));WRITE_ONCE (rdp->nocb_cb_sleep, true); + +if (rcu_segcblist_test_flags (cblist,SEGCBLIST_OFFLOADED )) { +if (rcu_segcblist_ready_cbs (cblist)) +WRITE_ONCE (rdp->nocb_cb_sleep, false); + }else { +/* +* De-offloading. Clear our flag and notify the de-offload worker. +* We won't touch the callbacks and keep sleeping until we ever +* get re-offloaded. +*/ +WARN_ON_ONCE (!rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB )); +rcu_segcblist_clear_flags (cblist,SEGCBLIST_KTHREAD_CB ); +if (!rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_GP )) + needwake_state = true; + } + +if (rdp->nocb_cb_sleep) +trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("CBSleep" )); +rcu_nocb_unlock_irqrestore (rdp, flags);if (needwake_gp)rcu_gp_kthread_wake (); -swait_event_interruptible_exclusive (rdp->nocb_cb_wq, - !READ_ONCE (rdp->nocb_cb_sleep)); -if (!smp_load_acquire (&rdp->nocb_cb_sleep)) {/* VVV */ + +if (needwake_state) +swake_up_one (&rdp->nocb_state_wq); + +do { +swait_event_interruptible_exclusive (rdp->nocb_cb_wq, +nocb_cb_wait_cond (rdp)); +/* ^^^ Ensure CB invocation follows _sleep test. */ -return ; - } -WARN_ON (signal_pending (current)); -trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("WokeEmpty" )); +if (smp_load_acquire (&rdp->nocb_cb_sleep)) { +WARN_ON (signal_pending (current)); +trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("WokeEmpty" )); + } + }while (!nocb_cb_can_run (rdp)); }/* @@ -2187,6 +2218,69 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)do_nocb_deferred_wakeup_common (rdp); } +static int __rcu_nocb_rdp_deoffload (struct rcu_data *rdp)
kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2231
Quoted 10 lines
kernel/rcu/tree_plugin.h +10
-0+{ +struct rcu_segcblist *cblist = &rdp->cblist; +struct rcu_node *rnp = rdp->mynode; +bool wake_cb = false; +unsigned long flags; + +printk ("De-offloading %d\n" , rdp->cpu); + +rcu_nocb_lock_irqsave (rdp, flags); +raw_spin_lock_rcu_node (rnp);
Could you explain why both nocb_lock and node lock are needed here? Besides rcu_nocb_{un}lock_* can skip the lock part based on the offload state, which gets modified right here (in a rcu_nocb_{un}lock_* critical secion), and I think this is error-prone, because you may get a unpaired lock-unlock (not in this patch though). Maybe just use node lock?
Quoted 13 lines
kernel/rcu/tree_plugin.h +13
-0+rcu_segcblist_offload (cblist, false); +raw_spin_unlock_rcu_node (rnp); + +if (rdp->nocb_cb_sleep) { + rdp->nocb_cb_sleep = false; + wake_cb = true; + } +rcu_nocb_unlock_irqrestore (rdp, flags); + +if (wake_cb) +swake_up_one (&rdp->nocb_cb_wq); + + swait_event_exclusive(rdp->nocb_state_wq,
kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2254
Quoted 10 lines
kernel/rcu/tree_plugin.h +10
-0+ !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)); + +return 0 ; +} + +static long rcu_nocb_rdp_deoffload (void *arg) +{ +struct rcu_data *rdp = arg; + +WARN_ON_ONCE (rdp->cpu !=raw_smp_processor_id ());
I think this warning can actually happen, if I understand how workqueue works correctly. Consider that the corresponding cpu gets offlined right after the rcu_nocb_cpu_deoffloaed(), and the workqueue of that cpu becomes unbound, and IIUC, workqueues don't do migration during cpu-offlining, which means the worker can be scheduled to other CPUs, and the work gets executed on another cpu. Am I missing something here?.
Regards,
Boqun
Quoted 53 lines
kernel/rcu/tree_plugin.h +32
-1+return __rcu_nocb_rdp_deoffload (rdp); +} + +int rcu_nocb_cpu_deoffload (int cpu) +{ +struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu); +int ret =0 ; + +if (rdp == rdp->nocb_gp_rdp) { +pr_info ("Can't deoffload an rdp GP leader (yet)\n" ); +return -EINVAL ; + } +mutex_lock (&rcu_state.barrier_mutex); +cpus_read_lock (); +if (rcu_segcblist_is_offloaded (&rdp->cblist)) { +if (cpu_online (cpu)) { + ret =work_on_cpu (cpu, rcu_nocb_rdp_deoffload, rdp); + }else { + ret =__rcu_nocb_rdp_deoffload (rdp); + } +if (!ret) +cpumask_clear_cpu (cpu, rcu_nocb_mask); + } +cpus_read_unlock (); +mutex_unlock (&rcu_state.barrier_mutex); + +return ret; +} +void __init rcu_init_nohz (void ) {int cpu;@@ -2229,7 +2323,8 @@ void __init rcu_init_nohz(void) rdp =per_cpu_ptr (&rcu_data, cpu);if (rcu_segcblist_empty (&rdp->cblist))rcu_segcblist_init (&rdp->cblist); -rcu_segcblist_offload (&rdp->cblist); +rcu_segcblist_offload (&rdp->cblist, true); +rcu_segcblist_set_flags (&rdp->cblist,SEGCBLIST_KTHREAD_CB ); }rcu_organize_nocb_kthreads (); }@@ -2239,6 +2334,7 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp) {init_swait_queue_head (&rdp->nocb_cb_wq);init_swait_queue_head (&rdp->nocb_gp_wq); +init_swait_queue_head (&rdp->nocb_state_wq);raw_spin_lock_init (&rdp->nocb_lock);raw_spin_lock_init (&rdp->nocb_bypass_lock);raw_spin_lock_init (&rdp->nocb_gp_lock);Signature · 1 line
2.25.1
Frederic Weisbecker
maintainerRe: [PATCH 05/16] rcu: De-offloading CB kthread
Quoted 4 lines · Boqun Feng
Hi Frederic,
Could you copy the [email protected] if you have another version, it will help RCU hobbyists like me to catch up news in RCU, thanks! ;-)
Sure! Will do!
Quoted 5 lines
Quoted 5 lines
kernel/rcu/tree_plugin.h +5
-0+static int __rcu_nocb_rdp_deoffload (struct rcu_data *rdp) +{ +struct rcu_segcblist *cblist = &rdp->cblist; +struct rcu_node *rnp = rdp->mynode; +bool wake_cb = false;
Quoted 8 lines
kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2231
Quoted 6 lines
kernel/rcu/tree_plugin.h +6
-0+unsigned long flags; + +printk ("De-offloading %d\n" , rdp->cpu); + +rcu_nocb_lock_irqsave (rdp, flags); +raw_spin_lock_rcu_node (rnp);Could you explain why both
nocb_lockand node lock are needed here?
So here is the common scheme with rcu nocb locking:
rcu_nocb_lock(rdp) {
if (rcu_segcblist_is_offloaded(rdp->cblist))
raw_spin_lock(rdp->nocb_lock) }
....
rcu_nocb_unlock(rdp) {
if (rcu_segcblist_is_offloaded(rdp->cblist))
raw_spin_unlock(rdp->nocb_lock) }
Here we first need to check rdp->cblist offloaded state locklessly. This is dangerous because we now can switch that offloaded state remotely. So we may run some code under rcu_nocb_lock() without actually locking while at the same time we switch to offloaded mode and we must run things under the lock.
Fortunately, rcu_nocb_lock(), and rcu_segcblist_is_offloaded() in general, is only called under either of the following situations (for which I need to add debug assertions, but that will be in a subsequent patchset):
1) hotplug write lock held
2) rcu_barrier mutex held
3) nocb lock held
4) rnp lock held
5) rdp is local
6) we run the nocb timer
1) and 2) are covered because we hold them while calling work_on_cpu().
3) and 4) are held when we change the flags of the rdp->cblist.
5) is covered by the fact we are modifying the rdp->cblist state only locally (using work_on_cpu()) with irqs disabled.
6) we cancel/rearm the timer before it can see half states.
But you make me realize one thing: we must protect against these situations especially when we switch the flags:
a) From SEGCBLIST_SOFTIRQ_ONLY to SEGCBLIST_OFFLOADED (switch from no locking to locking)
b) From 0 to SEGCBLIST_SOFTIRQ_ONLY (switch from locking to no locking)
Those are the critical state changes that modify the locking behaviour and we have to hold both rdp->nocb_lock and the rnp lock on these state changes.
Unfortunately in the case b) I omitted the rnp lock. So I need to fix that.
Quoted 4 lines
Besides rcu_nocb_{un}lock_* can skip the lock part based on the offload state, which gets modified right here (in a rcu_nocb_{un}lock_* critical secion), and I think this is error-prone, because you may get a unpaired lock-unlock (not in this patch though). Maybe just use node lock?
So that's why case 1 to 6 must be covered.
Quoted 23 lines
Quoted 23 lines
kernel/rcu/tree_plugin.h +22
-0+rcu_segcblist_offload (cblist, false); +raw_spin_unlock_rcu_node (rnp); + +if (rdp->nocb_cb_sleep) { + rdp->nocb_cb_sleep = false; + wake_cb = true; + } +rcu_nocb_unlock_irqrestore (rdp, flags); + +if (wake_cb) +swake_up_one (&rdp->nocb_cb_wq); + +swait_event_exclusive (rdp->nocb_state_wq, + !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB )); + +return 0 ; +} + +static long rcu_nocb_rdp_deoffload (void *arg) +{ +struct rcu_data *rdp = arg; +
Quoted 8 lines
kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2254
Quoted 1 line
kernel/rcu/tree_plugin.h +1
-0+WARN_ON_ONCE (rdp->cpu !=raw_smp_processor_id ());I think this warning can actually happen, if I understand how workqueue works correctly. Consider that the corresponding cpu gets offlined right after the
rcu_nocb_cpu_deoffloaed(), and the workqueue of that cpu becomes unbound, and IIUC, workqueues don't do migration during cpu-offlining, which means the worker can be scheduled to other CPUs, and the work gets executed on another cpu. Am I missing something here?.
We are holding cpus_read_lock() in rcu_nocb_cpu_offload(), this should prevent from that.
Thanks!
Re: [PATCH 05/16] rcu: De-offloading CB kthread
On Wed, Nov 04, 2020 at 03:31:35PM +0100, Frederic Weisbecker wrote:
[...]
Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
kernel/rcu/tree_plugin.h +22
-0+rcu_segcblist_offload (cblist, false); +raw_spin_unlock_rcu_node (rnp); + +if (rdp->nocb_cb_sleep) { + rdp->nocb_cb_sleep = false; + wake_cb = true; + } +rcu_nocb_unlock_irqrestore (rdp, flags); + +if (wake_cb) +swake_up_one (&rdp->nocb_cb_wq); + +swait_event_exclusive (rdp->nocb_state_wq, + !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB )); + +return 0 ; +} + +static long rcu_nocb_rdp_deoffload (void *arg) +{ +struct rcu_data *rdp = arg; +Quoted 8 lines
kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2254
Quoted 1 line
kernel/rcu/tree_plugin.h +1
-0+WARN_ON_ONCE (rdp->cpu !=raw_smp_processor_id ());I think this warning can actually happen, if I understand how workqueue works correctly. Consider that the corresponding cpu gets offlined right after the
rcu_nocb_cpu_deoffloaed(), and the workqueue of that cpu becomes unbound, and IIUC, workqueues don't do migration during cpu-offlining, which means the worker can be scheduled to other CPUs, and the work gets executed on another cpu. Am I missing something here?.
Quoted 3 lines
We are holding
cpus_read_lock()inrcu_nocb_cpu_offload(), this should prevent from that.
But what if the work doesn't get executed until we cpus_read_unlock() and someone offlines that CPU?
Regards,
Boqun
Quoted 1 line
Thanks!
Frederic Weisbecker
maintainerRe: [PATCH 05/16] rcu: De-offloading CB kthread
Quoted 34 lines · Boqun Feng
On Wed, Nov 04, 2020 at 03:31:35PM +0100, Frederic Weisbecker wrote:
[...]Quoted 32 lines
Quoted 31 lines
Quoted 14 lines
kernel/rcu/tree_plugin.h +13
-0+rcu_segcblist_offload (cblist, false); +raw_spin_unlock_rcu_node (rnp); + +if (rdp->nocb_cb_sleep) { + rdp->nocb_cb_sleep = false; + wake_cb = true; + } +rcu_nocb_unlock_irqrestore (rdp, flags); + +if (wake_cb) +swake_up_one (&rdp->nocb_cb_wq); + + swait_event_exclusive(rdp->nocb_state_wq,kernel/rcu/tree_plugin.h static void do_nocb_deferred_wakeup(struct rcu_data *rdp) · line 2254
Quoted 10 lines
kernel/rcu/tree_plugin.h +10
-0+ !rcu_segcblist_test_flags(cblist, SEGCBLIST_KTHREAD_CB)); + +return 0 ; +} + +static long rcu_nocb_rdp_deoffload (void *arg) +{ +struct rcu_data *rdp = arg; + +WARN_ON_ONCE (rdp->cpu !=raw_smp_processor_id ());I think this warning can actually happen, if I understand how workqueue works correctly. Consider that the corresponding cpu gets offlined right after the
rcu_nocb_cpu_deoffloaed(), and the workqueue of that cpu becomes unbound, and IIUC, workqueues don't do migration during cpu-offlining, which means the worker can be scheduled to other CPUs, and the work gets executed on another cpu. Am I missing something here?.
Quoted 6 lines
Quoted 3 lines
We are holding
cpus_read_lock()inrcu_nocb_cpu_offload(), this should prevent from that.But what if the work doesn't get executed until we
cpus_read_unlock()and someone offlines that CPU?
work_on_cpu() waits for completion before returning.
Quoted 4 lines
Regards,
BoqunQuoted 1 line
Thanks!