Provide runtime switching of offloaded CPUs
Subsystems (7)
Paul E. McKenney
maintainercover Provide runtime switching of offloaded CPUs
Hello!
Currently, it is necessary to reboot to change a given CPU between invoking its own RCU callbacks via softirq and offloading callback invocation to an rcuo kthread. This series allows CPUs that were initially offloaded to be toggled back and forth at runtime, without requiring a reboot.
Note that you can boot with all CPUs offloaded, de-offload them as needed immediately after boot, and thereafter toggled back and forth as desired.
1. Turn enabled/offload states into a common flag, courtesy of
Frederic Weisbecker.
2. Provide basic callback offloading state machine bits, courtesy
of Frederic Weisbecker.
3. Always init segcblist on CPU up, courtesy of Frederic Weisbecker.
4. De-offloading CB kthread, courtesy of Frederic Weisbecker.
5. Don't deoffload an offline CPU with pending work, courtesy of
Frederic Weisbecker.
6. De-offloading GP kthread, courtesy of Frederic Weisbecker.
7. Re-offload support, courtesy of Frederic Weisbecker.
8. Shutdown nocb timer on de-offloading, courtesy of Frederic
Weisbecker.
9. Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY, courtesy of
Frederic Weisbecker.
10. Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of
de-offloading, courtesy of Frederic Weisbecker.
11. Only cond_resched() from actual offloaded batch processing,
courtesy of Frederic Weisbecker.
12. Process batch locally as long as offloading isn't complete,
courtesy of Frederic Weisbecker.
13. Locally accelerate callbacks as long as offloading isn't complete,
courtesy of Frederic Weisbecker.
14. Add lockdep_is_cpus_held(), courtesy of Frederic Weisbecker.
15. Add timer_curr_running(), courtesy of Frederic Weisbecker.
16. Test runtime toggling of CPUs' callback offloading.
17. Support nocb toggle in TREE01, courtesy of Frederic Weisbecker.
18. Add grace period and task state to show_rcu_nocb_state() output.
19. Add nocb CB kthread list to show_rcu_nocb_state() output.
20. Code-style nits in callback-offloading toggling.
21. Do any deferred nocb wakeups at CPU offline time.
Thanx, Paul
------------------------------------------------------------------------
Documentation/admin-guide/kernel-parameters.txt | 8 include/linux/cpu.h | 2 include/linux/rcu_segcblist.h | 121 +++ include/linux/rcupdate.h | 4 include/linux/timer.h | 2 kernel/cpu.c | 7 kernel/rcu/rcu_segcblist.c | 17 kernel/rcu/rcu_segcblist.h | 79 +- kernel/rcu/rcutorture.c | 90 ++ kernel/rcu/tree.c | 31 kernel/rcu/tree.h | 2 kernel/rcu/tree_plugin.h | 433 +++++++++++-- kernel/time/timer.c | 13 tools/testing/selftests/rcutorture/configs/rcu/TREE01.boot | 4 14 files changed, 714 insertions(+), 99 deletions(-)
[email protected]
maintainerPATCH 1/21 rcu/nocb: Turn enabled/offload states into a common flag
From: Frederic Weisbecker <[email protected]>
This commit gathers the rcu_segcblist ->enabled and ->offloaded property field into a single ->flags bitmask to avoid further proliferation of individual u8 fields in the structure. This change prepares for the state formerly known as ->offloaded state to be modified at runtime.
--- include/linux/rcu_segcblist.h | 6 ++++-- kernel/rcu/rcu_segcblist.c | 6 +++--- kernel/rcu/rcu_segcblist.h | 23 +++++++++++++++++++++-- 3 files changed, 28 insertions(+), 7 deletions(-)
include/linux/rcu_segcblist.h +4 -2
@@ -63,6 +63,9 @@ struct rcu_cblist {#define RCU_NEXT_TAIL 3#define RCU_CBLIST_NSEGS 4 +#define SEGCBLIST_ENABLED BIT(0) +#define SEGCBLIST_OFFLOADED BIT(1) +struct rcu_segcblist {struct rcu_head *head;struct rcu_head **tails[RCU_CBLIST_NSEGS ];
@@ -73,8 +76,7 @@ struct rcu_segcblist {long len;#endif long seglen[RCU_CBLIST_NSEGS ]; -u8 enabled; -u8 offloaded; +u8 flags; };#define RCU_SEGCBLIST_INITIALIZER (n) \
kernel/rcu/rcu_segcblist.c +3 -3
@@ -246,7 +246,7 @@ void rcu_segcblist_init(struct rcu_segcblist *rsclp)rcu_segcblist_set_seglen (rsclp, i,0 ); }rcu_segcblist_set_len (rsclp,0 ); - rsclp->enabled =1 ; +rcu_segcblist_set_flags (rsclp,SEGCBLIST_ENABLED ); }/*
@@ -257,7 +257,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp) {WARN_ON_ONCE (!rcu_segcblist_empty (rsclp));WARN_ON_ONCE (rcu_segcblist_n_cbs (rsclp)); - rsclp->enabled =0 ; +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_ENABLED ); }/*
@@ -266,7 +266,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)*/ void rcu_segcblist_offload (struct rcu_segcblist *rsclp) { - rsclp->offloaded =1 ; +rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); }/*
kernel/rcu/rcu_segcblist.h +21 -2
@@ -53,19 +53,38 @@ static inline long rcu_segcblist_n_cbs(struct rcu_segcblist *rsclp)#endif } +static inline void rcu_segcblist_set_flags (struct rcu_segcblist *rsclp, +int flags) +{ + rsclp->flags |= flags; +} + +static inline void rcu_segcblist_clear_flags (struct rcu_segcblist *rsclp, +int flags) +{ + rsclp->flags &= ~flags; +} + +static inline bool rcu_segcblist_test_flags (struct rcu_segcblist *rsclp, +int flags) +{ +return READ_ONCE (rsclp->flags) & flags; +} +/* * Is the specified rcu_segcblist enabled, for example, not corresponding * to an offline CPU? */ static inline bool rcu_segcblist_is_enabled (struct rcu_segcblist *rsclp) { -return rsclp->enabled; +return rcu_segcblist_test_flags (rsclp,SEGCBLIST_ENABLED ); }/* Is the specified rcu_segcblist offloaded? */ static inline bool rcu_segcblist_is_offloaded (struct rcu_segcblist *rsclp) { -return IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && rsclp->offloaded; +return IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && +rcu_segcblist_test_flags (rsclp,SEGCBLIST_OFFLOADED ); }/*
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 2/21 rcu/nocb: Provide basic callback offloading state machine bits
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
Offloading and de-offloading RCU callback processes must be done carefully. There must never be a time at which callback processing is disabled because the task driving the offloading or de-offloading might be preempted or otherwise stalled at that point in time, which would result in OOM due to calbacks piling up indefinitely. This implies that there will be times during which a given CPU's callbacks might be concurrently invoked by both that CPU's RCU_SOFTIRQ handler (or, equivalently, that CPU's rcuc kthread) and by that CPU's rcuo kthread.
This situation could fatally confuse both rcu_barrier() and the CPU-hotplug offlining process, so these must be excluded during any concurrent-callback-invocation period. In addition, during times of concurrent callback invocation, changes to ->cblist must be protected both as needed for RCU_SOFTIRQ and as needed for the rcuo kthread.
This commit therefore defines and documents the states for a state machine that coordinates offloading and deoffloading.
--- include/linux/rcu_segcblist.h | 115 +++++++++++++++++++++++++++++++++++++++++- kernel/rcu/rcu_segcblist.c | 1 + kernel/rcu/rcu_segcblist.h | 12 ++++- kernel/rcu/tree.c | 3 ++ 4 files changed, 128 insertions(+), 3 deletions(-)
include/linux/rcu_segcblist.h +114 -1
@@ -63,8 +63,121 @@ struct rcu_cblist {#define RCU_NEXT_TAIL 3#define RCU_CBLIST_NSEGS 4 + +/* +* ==NOCB Offloading state machine== +* +* +* ---------------------------------------------------------------------------- +* | SEGCBLIST_SOFTIRQ_ONLY | +* | | +* | Callbacks processed by rcu_core() from softirqs or local | +* | rcuc kthread, without holding nocb_lock. | +* ---------------------------------------------------------------------------- +* | +* v +* ---------------------------------------------------------------------------- +* | SEGCBLIST_OFFLOADED | +* | | +* | Callbacks processed by rcu_core() from softirqs or local | +* | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads, | +* | allowing nocb_timer to be armed. | +* ---------------------------------------------------------------------------- +* | +* v +* ----------------------------------- +* | | +* v v +* --------------------------------------- ----------------------------------| +* | SEGCBLIST_OFFLOADED | | | SEGCBLIST_OFFLOADED | | +* | SEGCBLIST_KTHREAD_CB | | SEGCBLIST_KTHREAD_GP | +* | | | | +* | | | | +* | CB kthread woke up and | | GP kthread woke up and | +* | acknowledged SEGCBLIST_OFFLOADED. | | acknowledged SEGCBLIST_OFFLOADED| +* | Processes callbacks concurrently | | | +* | with rcu_core(), holding | | | +* | nocb_lock. | | | +* --------------------------------------- ----------------------------------- +* | | +* ----------------------------------- +* | +* v +* |--------------------------------------------------------------------------| +* | SEGCBLIST_OFFLOADED | | +* | SEGCBLIST_KTHREAD_CB | | +* | SEGCBLIST_KTHREAD_GP | +* | | +* | Kthreads handle callbacks holding nocb_lock, local rcu_core() stops | +* | handling callbacks. | +* ---------------------------------------------------------------------------- +*/ + + + +/* +* ==NOCB De-Offloading state machine== +* +* +* |--------------------------------------------------------------------------| +* | SEGCBLIST_OFFLOADED | | +* | SEGCBLIST_KTHREAD_CB | | +* | SEGCBLIST_KTHREAD_GP | +* | | +* | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() | +* | ignores callbacks. | +* ---------------------------------------------------------------------------- +* | +* v +* |--------------------------------------------------------------------------| +* | SEGCBLIST_KTHREAD_CB | | +* | SEGCBLIST_KTHREAD_GP | +* | | +* | CB/GP kthreads and local rcu_core() handle callbacks concurrently | +* | holding nocb_lock. Wake up CB and GP kthreads if necessary. | +* ---------------------------------------------------------------------------- +* | +* v +* ----------------------------------- +* | | +* v v +* ---------------------------------------------------------------------------| +* | | +* | SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP | +* | | | +* | GP kthread woke up and | CB kthread woke up and | +* | acknowledged the fact that | acknowledged the fact that | +* | SEGCBLIST_OFFLOADED got cleared. | SEGCBLIST_OFFLOADED got cleared. | +* | | The CB kthread goes to sleep | +* | The callbacks from the target CPU | until it ever gets re-offloaded. | +* | will be ignored from the GP kthread | | +* | loop. | | +* ---------------------------------------------------------------------------- +* | | +* ----------------------------------- +* | +* v +* ---------------------------------------------------------------------------- +* | 0 | +* | | +* | Callbacks processed by rcu_core() from softirqs or local | +* | rcuc kthread, while holding nocb_lock. Forbid nocb_timer to be armed. | +* | Flush pending nocb_timer. Flush nocb bypass callbacks. | +* ---------------------------------------------------------------------------- +* | +* v +* ---------------------------------------------------------------------------- +* | SEGCBLIST_SOFTIRQ_ONLY | +* | | +* | Callbacks processed by rcu_core() from softirqs or local | +* | rcuc kthread, without holding nocb_lock. | +* ---------------------------------------------------------------------------- +*/ #define SEGCBLIST_ENABLED BIT(0) -#define SEGCBLIST_OFFLOADED BIT(1) +#define SEGCBLIST_SOFTIRQ_ONLY BIT(1) +#define SEGCBLIST_KTHREAD_CB BIT(2) +#define SEGCBLIST_KTHREAD_GP BIT(3) +#define SEGCBLIST_OFFLOADED BIT(4)struct rcu_segcblist {struct rcu_head *head;
kernel/rcu/rcu_segcblist.c +1 -0
@@ -266,6 +266,7 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)*/ void rcu_segcblist_offload (struct rcu_segcblist *rsclp) { +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY );rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); }
kernel/rcu/rcu_segcblist.h +10 -2
@@ -83,8 +83,16 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)/* Is the specified rcu_segcblist offloaded? */ static inline bool rcu_segcblist_is_offloaded (struct rcu_segcblist *rsclp) { -return IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && -rcu_segcblist_test_flags (rsclp,SEGCBLIST_OFFLOADED ); +if (IS_ENABLED (CONFIG_RCU_NOCB_CPU )) { +/* +* Complete de-offloading happens only when SEGCBLIST_SOFTIRQ_ONLY +* is set. +*/ +if (!rcu_segcblist_test_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY )) +return true; + } + +return false; }/*
kernel/rcu/tree.c +3 -0
@@ -83,6 +83,9 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_data, rcu_data) = { .dynticks_nesting = 1, .dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE, .dynticks = ATOMIC_INIT(RCU_DYNTICK_CTRL_CTR), +#ifdef CONFIG_RCU_NOCB_CPU + .cblist.flags = SEGCBLIST_SOFTIRQ_ONLY, +#endif };static struct rcu_state rcu_state = { .level = { &rcu_state.node[0 ] },
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 3/21 rcu/nocb: Always init segcblist on CPU up
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
How the rdp->cblist enabled state is treated at CPU-hotplug time depends on whether or not that ->cblist is offloaded.
1) Not offloaded: The ->cblist is disabled when the CPU goes down. All its callbacks are migrated and none can to enqueued until after some later CPU-hotplug operation brings the CPU back up.
2) Offloaded: The ->cblist is not disabled on CPU down because the CB/GP kthreads must finish invoking the remaining callbacks. There is thus no need to re-enable it on CPU up.
Since the ->cblist offloaded state is set in stone at boot, it cannot change between CPU down and CPU up. So 1) and 2) are symmetrical.
However, given runtime toggling of the offloaded state, there are two additional asymmetrical scenarios:
3) The ->cblist is not offloaded when the CPU goes down. The ->cblist is later toggled to offloaded and then the CPU comes back up.
4) The ->cblist is offloaded when the CPU goes down. The ->cblist is later toggled to no longer be offloaded and then the CPU comes back up.
Scenario 4) is currently handled correctly. The ->cblist remains enabled on CPU down and gets re-initialized on CPU up. The toggling operation will wait until ->cblist is empty, so ->cblist will remain empty until CPU-up time.
The scenario 3) would run into trouble though, as the rdp is disabled on CPU down and not re-initialized/re-enabled on CPU up. Except that in this case, ->cblist is guaranteed to be empty because all its callbacks were migrated away at CPU-down time. And the CPU-up code already initializes and enables any empty ->cblist structures in order to handle the possibility of early-boot invocations of call_rcu() in the case where such invocations don't occur. So all that need be done is to adjust the locking.
--- kernel/rcu/tree.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
kernel/rcu/tree.c +9 -3
@@ -4015,12 +4015,18 @@ int rcutree_prepare_cpu(unsigned int cpu) rdp->qlen_last_fqs_check =0 ; rdp->n_force_qs_snap = rcu_state.n_force_qs; rdp->blimit = blimit; -if (rcu_segcblist_empty (&rdp->cblist) &&/* No early-boot CBs? */ - !rcu_segcblist_is_offloaded (&rdp->cblist)) -rcu_segcblist_init (&rdp->cblist);/* Re-enable callbacks. */ rdp->dynticks_nesting =1 ;/* CPU not up, no tearing. */ rcu_dynticks_eqs_online ();raw_spin_unlock_rcu_node (rnp);/* irqs remain disabled. */ +/* +* Lock in case the CB/GP kthreads are still around handling +* old callbacks (longer term we should flush all callbacks +* before completing CPU offline) +*/ +rcu_nocb_lock (rdp); +if (rcu_segcblist_empty (&rdp->cblist))/* No early-boot CBs? */ +rcu_segcblist_init (&rdp->cblist);/* Re-enable callbacks. */ +rcu_nocb_unlock (rdp);/* * Add CPU to leaf rcu_node pending-online bitmask. Any needed
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 4/21 rcu/nocb: De-offloading CB kthread
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
To de-offload callback processing back onto a CPU, it is necessary to clear SEGCBLIST_OFFLOAD and notify the nocb CB kthread, which will then clear its own bit flag and go to sleep to stop handling callbacks. This commit makes that change. It will also be necessary to notify the nocb GP kthread in this same way, which is the subject of a follow-on commit.
--- include/linux/rcupdate.h | 2 + kernel/rcu/rcu_segcblist.c | 10 ++-- kernel/rcu/rcu_segcblist.h | 2 +- kernel/rcu/tree.h | 1 + kernel/rcu/tree_plugin.h | 130 ++++++++++++++++++++++++++++++++++++++------- 5 files changed, 123 insertions(+), 22 deletions(-)
include/linux/rcupdate.h +2 -0
@@ -104,8 +104,10 @@ static inline void rcu_user_exit(void) { }#ifdef CONFIG_RCU_NOCB_CPU void rcu_init_nohz (void ); +int rcu_nocb_cpu_deoffload (int cpu);#else /* #ifdef CONFIG_RCU_NOCB_CPU */ static inline void rcu_init_nohz (void ) { } +static inline int rcu_nocb_cpu_deoffload (int cpu) {return 0 ; }#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ /**
kernel/rcu/rcu_segcblist.c +7 -3
@@ -264,10 +264,14 @@ void rcu_segcblist_disable(struct rcu_segcblist *rsclp)* Mark the specified rcu_segcblist structure as offloaded. This * structure must be empty. */ -void rcu_segcblist_offload (struct rcu_segcblist *rsclp) +void rcu_segcblist_offload (struct rcu_segcblist *rsclp,bool offload) { -rcu_segcblist_clear_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY ); -rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); +if (offload) { +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY ); +rcu_segcblist_set_flags (rsclp,SEGCBLIST_OFFLOADED ); + }else { +rcu_segcblist_clear_flags (rsclp,SEGCBLIST_OFFLOADED ); + } }/*
kernel/rcu/rcu_segcblist.h +1 -1
@@ -109,7 +109,7 @@ void rcu_segcblist_inc_len(struct rcu_segcblist *rsclp);void rcu_segcblist_add_len (struct rcu_segcblist *rsclp,long v);void rcu_segcblist_init (struct rcu_segcblist *rsclp);void rcu_segcblist_disable (struct rcu_segcblist *rsclp); -void rcu_segcblist_offload (struct rcu_segcblist *rsclp); +void rcu_segcblist_offload (struct rcu_segcblist *rsclp,bool offload);bool rcu_segcblist_ready_cbs (struct rcu_segcblist *rsclp);bool rcu_segcblist_pend_cbs (struct rcu_segcblist *rsclp);struct rcu_head *rcu_segcblist_first_cb (struct rcu_segcblist *rsclp);
kernel/rcu/tree.h +1 -0
@@ -201,6 +201,7 @@ struct rcu_data {/* 5) Callback offloading. */ #ifdef CONFIG_RCU_NOCB_CPU struct swait_queue_head nocb_cb_wq;/* For nocb kthreads to sleep on. */ +struct swait_queue_head nocb_state_wq;/* For offloading state changes */ struct task_struct *nocb_gp_kthread;raw_spinlock_t nocb_lock;/* Guard following pair of fields. */ atomic_t nocb_lock_contended;/* Contention experienced. */
kernel/rcu/tree_plugin.h +112 -18
179 lines not on this page · raw message
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 5/21 rcu/nocb: Don't deoffload an offline CPU with pending work
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
Offloaded CPUs do not migrate their callbacks, instead relying on their rcuo kthread to invoke them. But if the CPU is offline, it will be running neither its RCU_SOFTIRQ handler nor its rcuc kthread. This means that de-offloading an offline CPU that still has pending callbacks will strand those callbacks. This commit therefore refuses to toggle offline CPUs having pending callbacks.
--- kernel/rcu/tree_plugin.h | 9 +++++++++ 1 file changed, 9 insertions(+)
kernel/rcu/tree_plugin.h +9 -0
@@ -2227,6 +2227,15 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)printk ("De-offloading %d\n" , rdp->cpu);rcu_nocb_lock_irqsave (rdp, flags); +/* +* If there are still pending work offloaded, the offline +* CPU won't help much handling them. +*/ +if (cpu_is_offline (rdp->cpu) && !rcu_segcblist_empty (&rdp->cblist)) { +rcu_nocb_unlock_irqrestore (rdp, flags); +return -EBUSY ; + } +rcu_segcblist_offload (cblist, false);if (rdp->nocb_cb_sleep) {
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 6/21 rcu/nocb: De-offloading GP kthread
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
To de-offload callback processing back onto a CPU, it is necessary to clear SEGCBLIST_OFFLOAD and notify the nocb GP kthread, which will then clear its own bit flag and ignore this CPU until further notice. Whichever of the nocb CB and nocb GP kthreads is last to clear its own bit notifies the de-offloading worker kthread. Once notified, this worker kthread can proceed safe in the knowledge that the nocb CB and GP kthreads will no longer be manipulating this CPU's RCU callback list.
This commit makes this change.
--- kernel/rcu/tree_plugin.h | 54 +++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 51 insertions(+), 3 deletions(-)
kernel/rcu/tree_plugin.h +51 -3
@@ -1928,6 +1928,33 @@ static void do_nocb_bypass_wakeup_timer(struct timer_list *t)__call_rcu_nocb_wake (rdp, true, flags); } +static inline bool nocb_gp_enabled_cb (struct rcu_data *rdp) +{ +u8 flags =SEGCBLIST_OFFLOADED |SEGCBLIST_KTHREAD_GP ; + +return rcu_segcblist_test_flags (&rdp->cblist, flags); +} + +static inline bool nocb_gp_update_state (struct rcu_data *rdp,bool *needwake_state) +{ +struct rcu_segcblist *cblist = &rdp->cblist; + +if (rcu_segcblist_test_flags (cblist,SEGCBLIST_OFFLOADED )) { +return true; + }else { +/* +* De-offloading. Clear our flag and notify the de-offload worker. +* We will ignore this rdp until it ever gets re-offloaded. +*/ +WARN_ON_ONCE (!rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_GP )); +rcu_segcblist_clear_flags (cblist,SEGCBLIST_KTHREAD_GP ); +if (!rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB )) + *needwake_state = true; +return false; + } +} + +/* * No-CBs GP kthreads come here to wait for additional callbacks to show up * or for grace periods to end.
@@ -1956,8 +1983,17 @@ static void nocb_gp_wait(struct rcu_data *my_rdp)*/ WARN_ON_ONCE (my_rdp->nocb_gp_rdp != my_rdp);for (rdp = my_rdp; rdp; rdp = rdp->nocb_next_cb_rdp) { +bool needwake_state = false; +if (!nocb_gp_enabled_cb (rdp)) +continue ;trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("Check" ));rcu_nocb_lock_irqsave (rdp, flags); +if (!nocb_gp_update_state (rdp, &needwake_state)) { +rcu_nocb_unlock_irqrestore (rdp, flags); +if (needwake_state) +swake_up_one (&rdp->nocb_state_wq); +continue ; + } bypass_ncbs =rcu_cblist_n_cbs (&rdp->nocb_bypass); if (bypass_ncbs && (time_after(j, READ_ONCE(rdp->nocb_bypass_first) + 1) ||
@@ -2221,7 +2257,8 @@ static void do_nocb_deferred_wakeup(struct rcu_data *rdp)static int __rcu_nocb_rdp_deoffload (struct rcu_data *rdp) {struct rcu_segcblist *cblist = &rdp->cblist; -bool wake_cb = false; +struct rcu_data *rdp_gp = rdp->nocb_gp_rdp; +bool wake_cb = false, wake_gp = false;unsigned long flags;printk ("De-offloading %d\n" , rdp->cpu);
@@ -2247,9 +2284,19 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)if (wake_cb)swake_up_one (&rdp->nocb_cb_wq); -swait_event_exclusive (rdp->nocb_state_wq, - !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB )); +raw_spin_lock_irqsave (&rdp_gp->nocb_gp_lock, flags); +if (rdp_gp->nocb_gp_sleep) { + rdp_gp->nocb_gp_sleep = false; + wake_gp = true; + } +raw_spin_unlock_irqrestore (&rdp_gp->nocb_gp_lock, flags); +if (wake_gp) +wake_up_process (rdp_gp->nocb_gp_kthread); + +swait_event_exclusive (rdp->nocb_state_wq, + !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB | +SEGCBLIST_KTHREAD_GP ));return 0 ; }
@@ -2332,6 +2379,7 @@ void __init rcu_init_nohz(void)rcu_segcblist_init (&rdp->cblist);rcu_segcblist_offload (&rdp->cblist, true);rcu_segcblist_set_flags (&rdp->cblist,SEGCBLIST_KTHREAD_CB ); +rcu_segcblist_set_flags (&rdp->cblist,SEGCBLIST_KTHREAD_GP ); }rcu_organize_nocb_kthreads (); }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 7/21 rcu/nocb: Re-offload support
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
To re-offload the callback processing off of a CPU, it is necessary to clear SEGCBLIST_SOFTIRQ_ONLY, set SEGCBLIST_OFFLOADED, and then notify both the CB and GP kthreads so that they both set their own bit flag and start processing the callbacks remotely. The re-offloading worker is then notified that it can stop the RCU_SOFTIRQ handler (or rcuc kthread, as the case may be) from processing the callbacks locally.
Ordering must be carefully enforced so that the callbacks that used to be processed locally without locking will have have the same ordering properties when they are invoked by the nocb CB and GP kthreads.
This commit makes this change.
--- include/linux/rcupdate.h | 2 + kernel/rcu/tree_plugin.h | 158 ++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 138 insertions(+), 22 deletions(-)
include/linux/rcupdate.h +2 -0
@@ -104,9 +104,11 @@ static inline void rcu_user_exit(void) { }#ifdef CONFIG_RCU_NOCB_CPU void rcu_init_nohz (void ); +int rcu_nocb_cpu_offload (int cpu);int rcu_nocb_cpu_deoffload (int cpu);#else /* #ifdef CONFIG_RCU_NOCB_CPU */ static inline void rcu_init_nohz (void ) { } +static inline int rcu_nocb_cpu_offload (int cpu) {return -EINVAL ; }static inline int rcu_nocb_cpu_deoffload (int cpu) {return 0 ; }#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
kernel/rcu/tree_plugin.h +136 -22
217 lines not on this page · raw message
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 8/21 rcu/nocb: Shutdown nocb timer on de-offloading
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This commit ensures that the nocb timer is shut down before reaching the final de-offloaded state. The key goal is to prevent the timer handler from manipulating the callbacks without the protection of the nocb locks.
--- kernel/rcu/tree.h | 1 + kernel/rcu/tree_plugin.h | 12 +++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-)
kernel/rcu/tree.h +1 -0
@@ -257,6 +257,7 @@ struct rcu_data { };/* Values for nocb_defer_wakeup field in struct rcu_data. */ +#define RCU_NOCB_WAKE_OFF -1#define RCU_NOCB_WAKE_NOT 0#define RCU_NOCB_WAKE 1#define RCU_NOCB_WAKE_FORCE 2
kernel/rcu/tree_plugin.h +11 -1
@@ -1665,6 +1665,8 @@ static void wake_nocb_gp(struct rcu_data *rdp, bool force,static void wake_nocb_gp_defer (struct rcu_data *rdp,int waketype,const char *reason) { +if (rdp->nocb_defer_wakeup ==RCU_NOCB_WAKE_OFF ) +return ;if (rdp->nocb_defer_wakeup ==RCU_NOCB_WAKE_NOT )mod_timer (&rdp->nocb_timer, jiffies +1 );if (rdp->nocb_defer_wakeup < waketype)
@@ -2243,7 +2245,7 @@ static int rcu_nocb_cb_kthread(void *arg)/* Is a deferred wakeup of rcu_nocb_kthread() required? */ static int rcu_nocb_need_deferred_wakeup (struct rcu_data *rdp) { -return READ_ONCE (rdp->nocb_defer_wakeup); +return READ_ONCE (rdp->nocb_defer_wakeup) >RCU_NOCB_WAKE_NOT ; }/* Do a deferred wakeup of rcu_nocb_kthread(). */
@@ -2337,6 +2339,12 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)swait_event_exclusive (rdp->nocb_state_wq, !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP )); +/* Make sure nocb timer won't stay around */ +rcu_nocb_lock_irqsave (rdp, flags); +WRITE_ONCE (rdp->nocb_defer_wakeup,RCU_NOCB_WAKE_OFF ); +rcu_nocb_unlock_irqrestore (rdp, flags); +del_timer_sync (&rdp->nocb_timer); +return ret; }
@@ -2394,6 +2402,8 @@ static int __rcu_nocb_rdp_offload(struct rcu_data *rdp)* SEGCBLIST_SOFTIRQ_ONLY mode. */ raw_spin_lock_irqsave (&rdp->nocb_lock, flags); +/* Re-enable nocb timer */ +WRITE_ONCE (rdp->nocb_defer_wakeup,RCU_NOCB_WAKE_NOT );/* * We didn't take the nocb lock while working on the * rdp->cblist in SEGCBLIST_SOFTIRQ_ONLY mode.
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 9/21 rcu/nocb: Flush bypass before setting SEGCBLIST_SOFTIRQ_ONLY
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This commit flushes the bypass queue and sets state to avoid its being refilled before switching to the final de-offloaded state. To avoid refilling, this commit sets SEGCBLIST_SOFTIRQ_ONLY before re-enabling IRQs.
--- kernel/rcu/tree_plugin.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +10 -1
@@ -2339,12 +2339,21 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)swait_event_exclusive (rdp->nocb_state_wq, !rcu_segcblist_test_flags (cblist,SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP )); -/* Make sure nocb timer won't stay around */ rcu_nocb_lock_irqsave (rdp, flags); +/* Make sure nocb timer won't stay around */ WRITE_ONCE (rdp->nocb_defer_wakeup,RCU_NOCB_WAKE_OFF );rcu_nocb_unlock_irqrestore (rdp, flags);del_timer_sync (&rdp->nocb_timer); +/* +* Flush bypass. While IRQs are disabled and once we set +* SEGCBLIST_SOFTIRQ_ONLY, no callback is supposed to be +* enqueued on bypass. +*/ +rcu_nocb_lock_irqsave (rdp, flags); +rcu_nocb_flush_bypass (rdp,NULL , jiffies); +rcu_nocb_unlock_irqrestore (rdp, flags); +return ret; }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 10/21 rcu/nocb: Set SEGCBLIST_SOFTIRQ_ONLY at the very last stage of de-offloading
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This commit sets SEGCBLIST_SOFTIRQ_ONLY once toggling is otherwise fully complete, allowing further RCU callback manipulation to be carried out locklessly and locally.
--- kernel/rcu/tree_plugin.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/rcu/tree_plugin.h +8 -1
@@ -2352,7 +2352,14 @@ static int __rcu_nocb_rdp_deoffload(struct rcu_data *rdp)*/ rcu_nocb_lock_irqsave (rdp, flags);rcu_nocb_flush_bypass (rdp,NULL , jiffies); -rcu_nocb_unlock_irqrestore (rdp, flags); +rcu_segcblist_set_flags (cblist,SEGCBLIST_SOFTIRQ_ONLY ); +/* +* With SEGCBLIST_SOFTIRQ_ONLY, we can't use +* rcu_nocb_unlock_irqrestore() anymore. Theoretically we +* could set SEGCBLIST_SOFTIRQ_ONLY with cb unlocked and IRQs +* disabled now, but let's be paranoid. +*/ +raw_spin_unlock_irqrestore (&rdp->nocb_lock, flags);return ret; }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 11/21 rcu/nocb: Only cond_resched() from actual offloaded batch processing
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
During a toggle operations, rcu_do_batch() may be invoked concurrently by softirqs and offloaded processing for a given CPU's callbacks. This commit therefore makes sure cond_resched() is invoked only from the offloaded context.
--- kernel/rcu/tree.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/rcu/tree.c +1 -2
@@ -2516,8 +2516,7 @@ static void rcu_do_batch(struct rcu_data *rdp)/* Exceeded the time limit, so leave. */ break ; } -if (offloaded) { -WARN_ON_ONCE (in_serving_softirq ()); +if (!in_serving_softirq ()) {local_bh_enable ();lockdep_assert_irqs_enabled ();cond_resched_tasks_rcu_qs ();
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 12/21 rcu/nocb: Process batch locally as long as offloading isn't complete
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This commit makes sure to process the callbacks locally (via either RCU_SOFTIRQ or the rcuc kthread) whenever the segcblist isn't entirely offloaded. This ensures that callbacks are invoked one way or another while a CPU is in the middle of a toggle operation.
--- kernel/rcu/rcu_segcblist.h | 12 ++++++++++++ kernel/rcu/tree.c | 3 ++- 2 files changed, 14 insertions(+), 1 deletion(-)
kernel/rcu/rcu_segcblist.h +12 -0
@@ -95,6 +95,18 @@ static inline bool rcu_segcblist_is_offloaded(struct rcu_segcblist *rsclp)return false; } +static inline bool rcu_segcblist_completely_offloaded (struct rcu_segcblist *rsclp) +{ +int flags =SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP |SEGCBLIST_OFFLOADED ; + +if (IS_ENABLED (CONFIG_RCU_NOCB_CPU )) { +if ((rsclp->flags & flags) == flags) +return true; + } + +return false; +} +/* * Are all segments following the specified segment of the specified * rcu_segcblist structure empty of callbacks? (The specified
kernel/rcu/tree.c +2 -1
@@ -2700,6 +2700,7 @@ static __latent_entropy void rcu_core(void)struct rcu_data *rdp =raw_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode;const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist); +const bool do_batch = !rcu_segcblist_completely_offloaded (&rdp->cblist);if (cpu_is_offline (smp_processor_id ()))return ;
@@ -2729,7 +2730,7 @@ static __latent_entropy void rcu_core(void)rcu_check_gp_start_stall (rnp, rdp,rcu_jiffies_till_stall_check ());/* If there are callbacks ready, invoke them. */ -if (!offloaded &&rcu_segcblist_ready_cbs (&rdp->cblist) && +if (do_batch &&rcu_segcblist_ready_cbs (&rdp->cblist) &&likely (READ_ONCE (rcu_scheduler_fully_active)))rcu_do_batch (rdp);
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 13/21 rcu/nocb: Locally accelerate callbacks as long as offloading isn't complete
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
The local callbacks processing checks if any callbacks need acceleration. This commit carries out this checking under nocb lock protection in the middle of toggle operations, during which time rcu_core() executes concurrently with GP/CB kthreads.
--- kernel/rcu/tree.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
kernel/rcu/tree.c +3 -4
@@ -2699,7 +2699,6 @@ static __latent_entropy void rcu_core(void)unsigned long flags;struct rcu_data *rdp =raw_cpu_ptr (&rcu_data);struct rcu_node *rnp = rdp->mynode; -const bool offloaded =rcu_segcblist_is_offloaded (&rdp->cblist);const bool do_batch = !rcu_segcblist_completely_offloaded (&rdp->cblist);if (cpu_is_offline (smp_processor_id ()))
@@ -2720,11 +2719,11 @@ static __latent_entropy void rcu_core(void)/* No grace period and unregistered callbacks? */ if (!rcu_gp_in_progress () && -rcu_segcblist_is_enabled (&rdp->cblist) && !offloaded) { -local_irq_save (flags); +rcu_segcblist_is_enabled (&rdp->cblist) && do_batch) { +rcu_nocb_lock_irqsave (rdp, flags);if (!rcu_segcblist_restempty (&rdp->cblist,RCU_NEXT_READY_TAIL ))rcu_accelerate_cbs_unlocked (rnp, rdp); -local_irq_restore (flags); +rcu_nocb_unlock_irqrestore (rdp, flags); }rcu_check_gp_start_stall (rnp, rdp,rcu_jiffies_till_stall_check ());
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 14/21 cpu/hotplug: Add lockdep_is_cpus_held()
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This commit adds a lockdep_is_cpus_held() function to verify that the proper locks are held and that various operations are running in the correct context.
--- include/linux/cpu.h | 2 ++ kernel/cpu.c | 7 +++++++ 2 files changed, 9 insertions(+)
include/linux/cpu.h +2 -0
@@ -111,6 +111,8 @@ static inline void cpu_maps_update_done(void)#endif /* CONFIG_SMP */ extern struct bus_type cpu_subsys; +extern int lockdep_is_cpus_held (void ); +#ifdef CONFIG_HOTPLUG_CPU extern void cpus_write_lock (void );extern void cpus_write_unlock (void );
kernel/cpu.c +7 -0
@@ -330,6 +330,13 @@ void lockdep_assert_cpus_held(void)percpu_rwsem_assert_held (&cpu_hotplug_lock); } +#ifdef CONFIG_LOCKDEP +int lockdep_is_cpus_held (void ) +{ +return percpu_rwsem_is_held (&cpu_hotplug_lock); +} +#endif +static void lockdep_acquire_cpus_lock (void ) {rwsem_acquire (&cpu_hotplug_lock.dep_map,0 ,0 , _THIS_IP_);
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 15/21 timer: Add timer_curr_running()
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This commit adds a timer_curr_running() function that verifies that the current code is running in the context of the specified timer's handler.
--- include/linux/timer.h | 2 ++ kernel/time/timer.c | 13 +++++++++++++ 2 files changed, 15 insertions(+)
include/linux/timer.h +2 -0
@@ -192,6 +192,8 @@ extern int try_to_del_timer_sync(struct timer_list *timer);#define del_singleshot_timer_sync (t) del_timer_sync(t) +extern bool timer_curr_running (struct timer_list *timer); +extern void init_timers (void );struct hrtimer ;extern enum hrtimer_restart it_real_fn (struct hrtimer *);
kernel/time/timer.c +13 -0
@@ -1237,6 +1237,19 @@ int try_to_del_timer_sync(struct timer_list *timer) }EXPORT_SYMBOL (try_to_del_timer_sync); +bool timer_curr_running (struct timer_list *timer) +{ +int i; + +for (i =0 ; i <NR_BASES ; i++) { +struct timer_base *base =this_cpu_ptr (&timer_bases[i]); +if (base->running_timer == timer) +return true; + } + +return false; +} +#ifdef CONFIG_PREEMPT_RT static __init void timer_base_init_expiry_lock (struct timer_base *base) {
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 16/21 rcutorture: Test runtime toggling of CPUs' callback offloading
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Frederic Weisbecker is adding the ability to change the rcu_nocbs state of CPUs at runtime, that is, to offload and deoffload their RCU callback processing without the need to reboot. As the old saying goes, "if it ain't tested, it don't work", so this commit therefore adds prototype rcutorture testing for this capability.
--- Documentation/admin-guide/kernel-parameters.txt | 8 +++ kernel/rcu/rcutorture.c | 90 ++++++++++++++++++++++++- 2 files changed, 95 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -4338,6 +4338,14 @@ stress RCU, they don't participate in the actual test, hence the "fake". + rcutorture.nocbs_nthreads= [KNL] + Set number of RCU callback-offload togglers. + Zero (the default) disables toggling. + + rcutorture.nocbs_toggle= [KNL] + Set the delay in milliseconds between successive + callback-offload toggling attempts. + rcutorture.nreaders= [KNL] Set number of RCU readers. The value -1 selects N-1, where N is the number of CPUs. A value
kernel/rcu/rcutorture.c +87 -3
148 lines not on this page · raw message
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 17/21 tools/rcutorture: Support nocb toggle in TREE01
In reply to Paul E. McKenneyFrom: Frederic Weisbecker <[email protected]>
This commit adds periodic toggling of 7 of 8 CPUs every second to TREE01 in order to test NOCB toggle code.
--- tools/testing/selftests/rcutorture/configs/rcu/TREE01.boot | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tools/testing/selftests/rcutorture/configs/rcu/TREE01.boot +3 -1
@@ -2,5 +2,7 @@ maxcpus=8 nr_cpus=43 rcutree.gp_preinit_delay=3 rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 -rcu_nocbs=0 +rcu_nocbs=0-1,3-7 +rcutorture.nocbs_nthreads=8 +rcutorture.nocbs_toggle=1000 rcutorture.fwd_progress=0
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 18/21 rcu/nocb: Add grace period and task state to show_rcu_nocb_state() output
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit improves debuggability by indicating which grace period each batch of nocb callbacks is waiting on and by showing the task state and last CPU for reach nocb kthread.
--- kernel/rcu/rcu_segcblist.h | 11 +++++++++++ kernel/rcu/tree_plugin.h | 39 ++++++++++++++++++++++++++++++++------- 2 files changed, 43 insertions(+), 7 deletions(-)
kernel/rcu/rcu_segcblist.h +11 -0
@@ -117,6 +117,17 @@ static inline bool rcu_segcblist_restempty(struct rcu_segcblist *rsclp, int seg)return !READ_ONCE (*READ_ONCE (rsclp->tails[seg])); } +/* +* Is the specified segment of the specified rcu_segcblist structure +* empty of callbacks? +*/ +static inline bool rcu_segcblist_segempty (struct rcu_segcblist *rsclp,int seg) +{ +if (seg ==RCU_DONE_TAIL ) +return &rsclp->head == rsclp->tails[RCU_DONE_TAIL ]; +return rsclp->tails[seg -1 ] == rsclp->tails[seg]; +} +void rcu_segcblist_inc_len (struct rcu_segcblist *rsclp);void rcu_segcblist_add_len (struct rcu_segcblist *rsclp,long v);void rcu_segcblist_init (struct rcu_segcblist *rsclp);
kernel/rcu/tree_plugin.h +32 -7
@@ -2672,6 +2672,19 @@ void rcu_bind_current_to_nocb(void) }EXPORT_SYMBOL_GPL (rcu_bind_current_to_nocb); +// The ->on_cpu field is available only in CONFIG_SMP=y, so... +#ifdef CONFIG_SMP +static char *show_rcu_should_be_on_cpu (struct task_struct *tsp) +{ +return tsp && tsp->state ==TASK_RUNNING && !tsp->on_cpu ?"!" :"" ; +} +#else // #ifdef CONFIG_SMP +static char *show_rcu_should_be_on_cpu (struct task_struct *tsp) +{ +return "" ; +} +#endif // #else #ifdef CONFIG_SMP +/* * Dump out nocb grace-period kthread state for the specified rcu_data * structure.
@@ -2680,7 +2693,7 @@ static void show_rcu_nocb_gp_state(struct rcu_data *rdp) {struct rcu_node *rnp = rdp->mynode; - pr_info("nocb GP %d %c%c%c%c%c%c %c[%c%c] %c%c:%ld rnp %d:%d %lu\n", + pr_info("nocb GP %d %c%c%c%c%c%c %c[%c%c] %c%c:%ld rnp %d:%d %lu %c CPU %d%s\n", rdp->cpu, "kK"[!!rdp->nocb_gp_kthread], "lL"[raw_spin_is_locked(&rdp->nocb_gp_lock)],
@@ -2694,12 +2707,17 @@ static void show_rcu_nocb_gp_state(struct rcu_data *rdp)".B" [!!rdp->nocb_gp_bypass],".G" [!!rdp->nocb_gp_gp], (long )rdp->nocb_gp_seq, - rnp->grplo, rnp->grphi, READ_ONCE(rdp->nocb_gp_loops)); + rnp->grplo, rnp->grphi,READ_ONCE (rdp->nocb_gp_loops), + rdp->nocb_gp_kthread ?task_state_to_char (rdp->nocb_gp_kthread) :'.' , + rdp->nocb_cb_kthread ? (int )task_cpu (rdp->nocb_gp_kthread) :-1 , + show_rcu_should_be_on_cpu(rdp->nocb_cb_kthread)); }/* Dump out nocb kthread state for the specified rcu_data structure. */ static void show_rcu_nocb_state (struct rcu_data *rdp) { +char bufw[20 ]; +char bufr[20 ];struct rcu_segcblist *rsclp = &rdp->cblist;bool waslocked;bool wastimer;
@@ -2708,7 +2726,9 @@ static void show_rcu_nocb_state(struct rcu_data *rdp)if (rdp->nocb_gp_rdp == rdp)show_rcu_nocb_gp_state (rdp); - pr_info(" CB %d->%d %c%c%c%c%c%c F%ld L%ld C%d %c%c%c%c%c q%ld\n", +sprintf (bufw,"%ld" , rsclp->gp_seq[RCU_WAIT_TAIL ]); +sprintf (bufr,"%ld" , rsclp->gp_seq[RCU_NEXT_READY_TAIL ]); + pr_info(" CB %d^%d->%d %c%c%c%c%c%c F%ld L%ld C%d %c%c%s%c%s%c%c q%ld %c CPU %d%s\n", rdp->cpu, rdp->nocb_gp_rdp->cpu, "kK"[!!rdp->nocb_cb_kthread], "bB"[raw_spin_is_locked(&rdp->nocb_bypass_lock)],
@@ -2720,11 +2740,16 @@ static void show_rcu_nocb_state(struct rcu_data *rdp) jiffies - rdp->nocb_nobypass_last, rdp->nocb_nobypass_count,".D" [rcu_segcblist_ready_cbs (rsclp)], -".W" [!rcu_segcblist_restempty (rsclp,RCU_DONE_TAIL )], -".R" [!rcu_segcblist_restempty (rsclp,RCU_WAIT_TAIL )], -".N" [!rcu_segcblist_restempty (rsclp,RCU_NEXT_READY_TAIL )], +".W" [!rcu_segcblist_segempty (rsclp,RCU_WAIT_TAIL )], +rcu_segcblist_segempty (rsclp,RCU_WAIT_TAIL ) ?"" : bufw, +".R" [!rcu_segcblist_segempty (rsclp,RCU_NEXT_READY_TAIL )], +rcu_segcblist_segempty (rsclp,RCU_NEXT_READY_TAIL ) ?"" : bufr, +".N" [!rcu_segcblist_segempty (rsclp,RCU_NEXT_TAIL )],".B" [!!rcu_cblist_n_cbs (&rdp->nocb_bypass)], - rcu_segcblist_n_cbs(&rdp->cblist)); +rcu_segcblist_n_cbs (&rdp->cblist), + rdp->nocb_cb_kthread ?task_state_to_char (rdp->nocb_cb_kthread) :'.' , + rdp->nocb_cb_kthread ? (int )task_cpu (rdp->nocb_gp_kthread) :-1 , + show_rcu_should_be_on_cpu(rdp->nocb_cb_kthread));/* It is OK for GP kthreads to have GP state. */ if (rdp->nocb_gp_rdp == rdp)
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 19/21 rcu/nocb: Add nocb CB kthread list to show_rcu_nocb_state() output
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit improves debuggability by indicating laying out the order in which rcuoc kthreads appear in the ->nocb_next_cb_rdp list.
--- kernel/rcu/tree_plugin.h | 1 + 1 file changed, 1 insertion(+)
kernel/rcu/tree_plugin.h +1 -0
@@ -2730,6 +2730,7 @@ static void show_rcu_nocb_state(struct rcu_data *rdp)sprintf (bufr,"%ld" , rsclp->gp_seq[RCU_NEXT_READY_TAIL ]); pr_info(" CB %d^%d->%d %c%c%c%c%c%c F%ld L%ld C%d %c%c%s%c%s%c%c q%ld %c CPU %d%s\n", rdp->cpu, rdp->nocb_gp_rdp->cpu, + rdp->nocb_next_cb_rdp ? rdp->nocb_next_cb_rdp->cpu : -1, "kK"[!!rdp->nocb_cb_kthread], "bB"[raw_spin_is_locked(&rdp->nocb_bypass_lock)], "cC"[!!atomic_read(&rdp->nocb_lock_contended)],
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 20/21 rcu/nocb: Code-style nits in callback-offloading toggling
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
This commit addresses a few code-style nits in callback-offloading toggling, including one that predates this toggling.
--- kernel/rcu/rcu_segcblist.h | 19 ++++++------------- kernel/rcu/tree_plugin.h | 10 +++++----- 2 files changed, 11 insertions(+), 18 deletions(-)
kernel/rcu/rcu_segcblist.h +6 -13
@@ -80,17 +80,12 @@ static inline bool rcu_segcblist_is_enabled(struct rcu_segcblist *rsclp)return rcu_segcblist_test_flags (rsclp,SEGCBLIST_ENABLED ); } -/* Is the specified rcu_segcblist offloaded? */ +/* Is the specified rcu_segcblist offloaded, or is SEGCBLIST_SOFTIRQ_ONLY set? */ static inline bool rcu_segcblist_is_offloaded (struct rcu_segcblist *rsclp) { -if (IS_ENABLED (CONFIG_RCU_NOCB_CPU )) { -/* -* Complete de-offloading happens only when SEGCBLIST_SOFTIRQ_ONLY -* is set. -*/ -if (!rcu_segcblist_test_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY )) -return true; - } +if (IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && + !rcu_segcblist_test_flags (rsclp,SEGCBLIST_SOFTIRQ_ONLY )) +return true;return false; }
@@ -99,10 +94,8 @@ static inline bool rcu_segcblist_completely_offloaded(struct rcu_segcblist *rscl {int flags =SEGCBLIST_KTHREAD_CB |SEGCBLIST_KTHREAD_GP |SEGCBLIST_OFFLOADED ; -if (IS_ENABLED (CONFIG_RCU_NOCB_CPU )) { -if ((rsclp->flags & flags) == flags) -return true; - } +if (IS_ENABLED (CONFIG_RCU_NOCB_CPU ) && (rsclp->flags & flags) == flags) +return true;return false; }
kernel/rcu/tree_plugin.h +5 -5
@@ -2160,11 +2160,11 @@ static inline bool nocb_cb_wait_cond(struct rcu_data *rdp)static void nocb_cb_wait (struct rcu_data *rdp) {struct rcu_segcblist *cblist = &rdp->cblist; -struct rcu_node *rnp = rdp->mynode; -bool needwake_state = false; -bool needwake_gp = false;unsigned long cur_gp_seq;unsigned long flags; +bool needwake_state = false; +bool needwake_gp = false; +struct rcu_node *rnp = rdp->mynode;local_irq_save (flags);rcu_momentary_dyntick_idle ();
@@ -2217,8 +2217,8 @@ static void nocb_cb_wait(struct rcu_data *rdp)swait_event_interruptible_exclusive (rdp->nocb_cb_wq,nocb_cb_wait_cond (rdp)); -/* ^^^ Ensure CB invocation follows _sleep test. */ -if (smp_load_acquire (&rdp->nocb_cb_sleep)) { +// VVV Ensure CB invocation follows _sleep test. +if (smp_load_acquire (&rdp->nocb_cb_sleep)) {// ^^^ WARN_ON (signal_pending (current));trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("WokeEmpty" )); }
Signature · 1 line
2.9.5
[email protected]
maintainerPATCH 21/21 rcu: Do any deferred nocb wakeups at CPU offline time
In reply to Paul E. McKenneyFrom: "Paul E. McKenney" <[email protected]>
Because the need to wake a nocb GP kthread ("rcuog") is sometimes detected when wakeups cannot be done, these wakeups can be deferred. The wakeups are then carried out by calls to do_nocb_deferred_wakeup() at various safe points in the code, including RCU's idle hooks. However, when a CPU goes offline, it invokes arch_cpu_idle_dead() without invoking any of RCU's idle hooks.
This commit therefore adds a call to do_nocb_deferred_wakeup() in rcu_report_dead() in order to handle any deferred wakeups that have been requested by the outgoing CPU.
--- kernel/rcu/tree.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/rcu/tree.c +3 -0
@@ -4178,6 +4178,9 @@ void rcu_report_dead(unsigned int cpu)struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu);struct rcu_node *rnp = rdp->mynode;/* Outgoing CPU's rdp & rnp. */ +// Do any dangling deferred wakeups. +do_nocb_deferred_wakeup (rdp); +/* QS for any half-done expedited grace period. */ preempt_disable ();rcu_report_exp_rdp (this_cpu_ptr (&rcu_data));
Signature · 1 line
2.9.5