hrtimer: Fix timers queued locally from offline CPUs
Subsystems (2)
Frederic Weisbecker
maintainerv3 cover hrtimer: Fix timers queued locally from offline CPUs
5c0930ccaad5 ("hrtimers: Push pending hrtimers away from outgoing CPU earlier") was introduced to fix stalls with scheduler bandwidth timers getting migrated while some kthreads handling CPU hotplug rely on bandwidth.
However this has introduced several other issues which used to be confined to RCU. But not anymore as it is spreading to hotplug code itself (https://lore.kernel.org/all/[email protected]/)
Instead of introducing yet another new hackery, fix the problem in hrtimers for everyone.
Changes since v2:
* Fix and add some tags
* Fix extra newlines
Frederic Weisbecker (3):
hrtimers: Force migrate away hrtimers queued after
CPUHP_AP_HRTIMERS_DYING
rcu: Remove swake_up_one_online() bandaid
Revert "rcu/nocb: Fix rcuog wake-up from offline softirq"
include/linux/hrtimer_defs.h | 1 + kernel/rcu/tree.c | 34 +---------------------- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_nocb.h | 10 ++----- kernel/time/hrtimer.c | 53 +++++++++++++++++++++++++++++++++--- 5 files changed, 54 insertions(+), 46 deletions(-)
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH v3 1/3 hrtimers: Force migrate away hrtimers queued after CPUHP_AP_HRTIMERS_DYING
hrtimers are migrated away from the dying CPU to any online target at the CPUHP_AP_HRTIMERS_DYING stage in order not to delay bandwidth timers handling tasks involved in the CPU hotplug forward progress.
However wake ups can still be performed by the outgoing CPU after CPUHP_AP_HRTIMERS_DYING. Those can result again in bandwidth timers being armed. Depending on several considerations (crystal ball power management based election, earliest timer already enqueued, timer migration enabled or not), the target may eventually be the current CPU even if offline. If that happens, the timer is eventually ignored.
The most notable example is RCU which had to deal with each an every of those wake-ups by deferring them to an online CPU, along with related workarounds:
_ e787644caf76 (rcu: Defer RCU kthreads wakeup when CPU is dying) _ 9139f93209d1 (rcu/nocb: Fix RT throttling hrtimer armed from offline CPU) _ f7345ccc62a4 (rcu/nocb: Fix rcuog wake-up from offline softirq)
The problem isn't confined to RCU though as the stop machine kthread (which runs CPUHP_AP_HRTIMERS_DYING) reports its completion at the end and performs a wake up that eventually arms the deadline server timer:
WARNING: CPU: 94 PID: 588 at kernel/time/hrtimer.c:1086 hrtimer_start_range_ns+0x289/0x2d0
Modules linked in:
CPU: 94 UID: 0 PID: 588 Comm: migration/94 Not tainted
Stopper: multi_cpu_stop+0x0/0x120 <- stop_machine_cpuslocked+0x66/0xc0
RIP: 0010:hrtimer_start_range_ns+0x289/0x2d0
Code: 41 5c 41 5d 41 5e 41 5f 5d e9 63 94 ea 00 0f 0b 48 83 c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d e9 39 fc 15 01 0f 0b e9 c1 fd ff ff <0f> 0b 48 8b
+45 00 e9 59 ff ff ff f3 0f 1e fa 65 8b 05 1d ec e8 7e
RSP: 0018:ffffc900019cbcc8 EFLAGS: 00010046
RAX: ffff88bf449a4c40 RBX: 0000000000000082 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffff88bf43224c80 RDI: ffff88bf449a4c40
RBP: ffff88bf449a4c80 R08: ffff888280970090 R09: 0000000000000000
R10: ffff88bf432252e0 R11: ffffffff811abf70 R12: ffff88bf449a4c40
R13: ffff88bf43234b28 R14: ffff88bf43224c80 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff88bf44980000(0000) knlGS:0000000000000000
CR2: 0000000000000000 CR3: 000000404b230001 CR4: 0000000000770ef0
PKRU: 55555554
Call Trace:
<TASK>
? __warn+0xcf/0x1b0
? hrtimer_start_range_ns+0x289/0x2d0
? report_bug+0x120/0x1a0
? handle_bug+0x60/0x90
? exc_invalid_op+0x1a/0x50
? asm_exc_invalid_op+0x1a/0x20
? register_refined_jiffies+0xb0/0xb0
? hrtimer_start_range_ns+0x289/0x2d0
? hrtimer_start_range_ns+0x186/0x2d0
start_dl_timer+0xfc/0x150
enqueue_dl_entity+0x367/0x640
dl_server_start+0x53/0xa0
enqueue_task_fair+0x363/0x460
enqueue_task+0x3c/0x200
ttwu_do_activate+0x94/0x240
try_to_wake_up+0x315/0x600
complete+0x4b/0x80Instead of providing yet another bandaid to work around the situation, fix it from hrtimers infrastructure instead: always migrate away a timer to an online target whenever it is enqueued from an offline CPU.
This will also allow to revert all the above RCU disgraceful hacks.
--- include/linux/hrtimer_defs.h | 1 + kernel/time/hrtimer.c | 53 +++++++++++++++++++++++++++++++++--- 2 files changed, 50 insertions(+), 4 deletions(-)
include/linux/hrtimer_defs.h +1 -0
@@ -125,6 +125,7 @@ struct hrtimer_cpu_base {ktime_t softirq_expires_next;struct hrtimer *softirq_next_timer;struct hrtimer_clock_base clock_base[HRTIMER_MAX_CLOCK_BASES ]; +call_single_data_t csd; }____cacheline_aligned ;
kernel/time/hrtimer.c +49 -4
93 lines not on this page · raw message
Signature · 1 line
2.46.0
Re: [PATCH 1/3 v3] hrtimers: Force migrate away hrtimers queued after CPUHP_AP_HRTIMERS_DYING
Quoted 15 lines · Frederic Weisbecker
hrtimers are migrated away from the dying CPU to any online target at the
CPUHP_AP_HRTIMERS_DYINGstage in order not to delay bandwidth timers handling tasks involved in the CPU hotplug forward progress.However wake ups can still be performed by the outgoing CPU after
CPUHP_AP_HRTIMERS_DYING. Those can result again in bandwidth timers being armed. Depending on several considerations (crystal ball power management based election, earliest timer already enqueued, timer migration enabled or not), the target may eventually be the current CPU even if offline. If that happens, the timer is eventually ignored.The most notable example is RCU which had to deal with each an every of those wake-ups by deferring them to an online CPU, along with related workarounds:
Quoted 7 lines
_
e787644caf76(rcu: Defer RCU kthreads wakeup when CPU is dying) _9139f93209d1(rcu/nocb: Fix RT throttling hrtimer armed from offline CPU) _f7345ccc62a4(rcu/nocb: Fix rcuog wake-up from offline softirq)The problem isn't confined to RCU though as the stop machine kthread (which runs
CPUHP_AP_HRTIMERS_DYING) reports its completion at the end and performs a wake up that eventually arms the deadline server timer:
Where does it report the completion?
Quoted 8 lines
WARNING: CPU: 94 PID: 588 at kernel/time/hrtimer.c:1086 hrtimer_start_range_ns+0x289/0x2d0
CPU: 94 UID: 0 PID: 588 Comm: migration/94 Not tainted
Stopper: multi_cpu_stop+0x0/0x120 <- stop_machine_cpuslocked+0x66/0xc0
RIP: 0010:hrtimer_start_range_ns+0x289/0x2d0
Call Trace:
? __warn+0xcf/0x1b0
? hrtimer_start_range_ns+0x289/0x2d0
? hrtimer_start_range_ns+0x186/0x2d0Quoted 8 lines
start_dl_timer+0xfc/0x150
enqueue_dl_entity+0x367/0x640
dl_server_start+0x53/0xa0
enqueue_task_fair+0x363/0x460
enqueue_task+0x3c/0x200
ttwu_do_activate+0x94/0x240
try_to_wake_up+0x315/0x600
complete+0x4b/0x80You trimmed the backtrace at the wrong end. You left all the useless register gunk around, but removed the call chain leading up to the completion. :)
I assume it's the complete in stomp_machine() ....
Quoted 29 lines
Instead of providing yet another bandaid to work around the situation, fix it from hrtimers infrastructure instead: always migrate away a timer to an online target whenever it is enqueued from an offline CPU.
(quoted diff) +25
-0+/* + * If the current CPU is offline and timers have been already + * migrated away, make sure not to enqueue locally and perform + * a remote retrigger as a last resort. + */ +static void enqueue_hrtimer_offline(struct hrtimer *timer, + struct hrtimer_clock_base *base, + const enum hrtimer_mode mode) +{ +#ifdef CONFIG_HOTPLUG_CPU + struct hrtimer_cpu_base *new_cpu_base, *old_cpu_base, *this_cpu_base; + struct hrtimer_clock_base *new_base; + int cpu; + + old_cpu_base = base->cpu_base; + this_cpu_base = this_cpu_ptr(&hrtimer_bases); + + if (old_cpu_base == this_cpu_base || !old_cpu_base->online) { + WARN_ON_ONCE(hrtimer_callback_running(timer)); + cpu = cpumask_any_and(cpu_online_mask, + housekeeping_cpumask(HK_TYPE_TIMER)); + new_cpu_base = &per_cpu(hrtimer_bases, cpu); + new_base = &new_cpu_base->clock_base[base->index]; + WRITE_ONCE(timer->base, &migration_base); + raw_spin_unlock(&old_cpu_base->lock);
Review
Quoted 10 lines
(quoted diff) +10
-0+ raw_spin_lock(&new_cpu_base->lock); + WRITE_ONCE(timer->base, new_base); + } else { + new_base = base; + new_cpu_base = new_base->cpu_base; + cpu = new_cpu_base->cpu; + } + + if (enqueue_hrtimer(timer, new_base, mode)) + smp_call_function_single_async(cpu, &new_cpu_base->csd);
Duh. This reimplementation of switch_hrtimer_base() is really aweful. We can be smarter than that. Untested patch below.
Thanks,
tglx
---
include/linux/hrtimer_defs.h +1 -0
@@ -125,6 +125,7 @@ struct hrtimer_cpu_base {ktime_t softirq_expires_next;struct hrtimer *softirq_next_timer;struct hrtimer_clock_base clock_base[HRTIMER_MAX_CLOCK_BASES ]; +call_single_data_t csd; }____cacheline_aligned ;
kernel/time/hrtimer.c +31 -7
@@ -58,6 +58,8 @@#define HRTIMER_ACTIVE_SOFT (HRTIMER_ACTIVE_HARD << MASK_SHIFT)#define HRTIMER_ACTIVE_ALL (HRTIMER_ACTIVE_SOFT | HRTIMER_ACTIVE_HARD) +static void retrigger_next_event (void *arg); +/* * The timer bases: *
@@ -111,7 +113,8 @@ DEFINE_PER_CPU(struct hrtimer_cpu_base, .clockid = CLOCK_TAI, .get_time = &ktime_get_clocktai, }, - } + }, + .csd =CSD_INIT (retrigger_next_event,NULL ) };static const int hrtimer_clock_to_base_table[MAX_CLOCKS ] = {
@@ -208,6 +211,13 @@ struct hrtimer_cpu_base *get_target_baseif (static_branch_likely (&timers_migration_enabled) && !pinned)return &per_cpu (hrtimer_bases,get_nohz_timer_target ());#endif +#ifdef CONFIG_HOTPLUG_CPU +if (unlikely (!base->online)) { +int cpu =cpumask_any_and (cpu_online_mask,housekeeping_cpumask (HK_TYPE_TIMER )); + +return &per_cpu (hrtimer_bases, cpu); + } +#endif return base; }
@@ -254,7 +264,7 @@ switch_hrtimer_base(struct hrtimer *timeraw_spin_unlock (&base->cpu_base->lock);raw_spin_lock (&new_base->cpu_base->lock); -if (new_cpu_base != this_cpu_base && +if (new_cpu_base != this_cpu_base && this_cpu_base->online &&hrtimer_check_target (timer, new_base)) {raw_spin_unlock (&new_base->cpu_base->lock);raw_spin_lock (&base->cpu_base->lock);
@@ -716,8 +726,6 @@ static inline int hrtimer_is_hres_enablereturn hrtimer_hres_enabled; } -static void retrigger_next_event (void *arg); -/* * Switch to high resolution mode */
@@ -1206,6 +1214,7 @@ static int __hrtimer_start_range_ns(stru u64 delta_ns, const enum hrtimer_mode mode, struct hrtimer_clock_base *base) { + struct hrtimer_cpu_base *this_cpu_base = this_cpu_ptr(&hrtimer_bases);struct hrtimer_clock_base *new_base;bool force_local, first;
@@ -1217,10 +1226,16 @@ static int __hrtimer_start_range_ns(stru* and enforce reprogramming after it is queued no matter whether * it is the new first expiring timer again or not. */ - force_local = base->cpu_base ==this_cpu_ptr (&hrtimer_bases); + force_local = base->cpu_base == this_cpu_base; force_local &= base->cpu_base->next_timer == timer;/* +* Don't force local queuing if this enqueue happens on a unplugged +* CPU after hrtimer_cpu_dying() has been invoked. +*/ + force_local &= this_cpu_base->online; + +/* * Remove an active timer from the queue. In case it is not queued * on the current CPU, make sure that remove_hrtimer() updates the * remote data correctly.
@@ -1249,8 +1264,17 @@ static int __hrtimer_start_range_ns(stru } first =enqueue_hrtimer (timer, new_base, mode); -if (!force_local) -return first; +if (!force_local) { +if (likely (this_cpu_base->online)) +return first; + +if (first) { +struct hrtimer_cpu_base *new_cpu_base = new_base->cpu_base; + +smp_call_function_single_async (new_cpu_base->cpu, &new_cpu_base->csd); + } +return 0 ; + }/* * Timer was forced to stay on the current CPU to avoid
Frederic Weisbecker
maintainerRe: [PATCH 1/3 v3] hrtimers: Force migrate away hrtimers queued after CPUHP_AP_HRTIMERS_DYING
Quoted 20 lines · Thomas Gleixner
Quoted 19 lines · Frederic Weisbecker
hrtimers are migrated away from the dying CPU to any online target at the
CPUHP_AP_HRTIMERS_DYINGstage in order not to delay bandwidth timers handling tasks involved in the CPU hotplug forward progress.However wake ups can still be performed by the outgoing CPU after
CPUHP_AP_HRTIMERS_DYING. Those can result again in bandwidth timers being armed. Depending on several considerations (crystal ball power management based election, earliest timer already enqueued, timer migration enabled or not), the target may eventually be the current CPU even if offline. If that happens, the timer is eventually ignored.The most notable example is RCU which had to deal with each an every of those wake-ups by deferring them to an online CPU, along with related workarounds:
_
e787644caf76(rcu: Defer RCU kthreads wakeup when CPU is dying) _9139f93209d1(rcu/nocb: Fix RT throttling hrtimer armed from offline CPU) _f7345ccc62a4(rcu/nocb: Fix rcuog wake-up from offline softirq)
Quoted 5 lines
Quoted 3 lines
The problem isn't confined to RCU though as the stop machine kthread (which runs
CPUHP_AP_HRTIMERS_DYING) reports its completion at the end and performs a wake up that eventually arms the deadline server timer:Where does it report the completion?
Right, that's cpu_stop_signal_done()
Quoted 18 lines
Quoted 17 lines
WARNING: CPU: 94 PID: 588 at kernel/time/hrtimer.c:1086 hrtimer_start_range_ns+0x289/0x2d0 CPU: 94 UID: 0 PID: 588 Comm: migration/94 Not tainted Stopper: multi_cpu_stop+0x0/0x120 <- stop_machine_cpuslocked+0x66/0xc0 RIP: 0010:hrtimer_start_range_ns+0x289/0x2d0 Call Trace: ? __warn+0xcf/0x1b0 ? hrtimer_start_range_ns+0x289/0x2d0 ? hrtimer_start_range_ns+0x186/0x2d0 start_dl_timer+0xfc/0x150 enqueue_dl_entity+0x367/0x640 dl_server_start+0x53/0xa0 enqueue_task_fair+0x363/0x460 enqueue_task+0x3c/0x200 ttwu_do_activate+0x94/0x240 try_to_wake_up+0x315/0x600 complete+0x4b/0x80
Quoted 5 lines
You trimmed the backtrace at the wrong end. You left all the useless register gunk around, but removed the call chain leading up to the completion. :)
I assume it's the complete in
stomp_machine()....
Bah! Indeed, lemme fix that...
Quoted 30 lines
Quoted 30 lines
(quoted diff) +30
-0+/* + * If the current CPU is offline and timers have been already + * migrated away, make sure not to enqueue locally and perform + * a remote retrigger as a last resort. + */ +static void enqueue_hrtimer_offline(struct hrtimer *timer, + struct hrtimer_clock_base *base, + const enum hrtimer_mode mode) +{ +#ifdef CONFIG_HOTPLUG_CPU + struct hrtimer_cpu_base *new_cpu_base, *old_cpu_base, *this_cpu_base; + struct hrtimer_clock_base *new_base; + int cpu; + + old_cpu_base = base->cpu_base; + this_cpu_base = this_cpu_ptr(&hrtimer_bases); + + if (old_cpu_base == this_cpu_base || !old_cpu_base->online) { + WARN_ON_ONCE(hrtimer_callback_running(timer)); + cpu = cpumask_any_and(cpu_online_mask, + housekeeping_cpumask(HK_TYPE_TIMER)); + new_cpu_base = &per_cpu(hrtimer_bases, cpu); + new_base = &new_cpu_base->clock_base[base->index]; + WRITE_ONCE(timer->base, &migration_base); + raw_spin_unlock(&old_cpu_base->lock); + raw_spin_lock(&new_cpu_base->lock); + WRITE_ONCE(timer->base, new_base); + } else { + new_base = base; + new_cpu_base = new_base->cpu_base;
Quoted 8 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ cpu = new_cpu_base->cpu; + } + + if (enqueue_hrtimer(timer, new_base, mode)) + smp_call_function_single_async(cpu, &new_cpu_base->csd);Duh. This reimplementation of
switch_hrtimer_base()is really aweful. We can be smarter than that. Untested patch below.
Indeed, and I tried to "fix" switch_hrtimer_base() but didn't managed to do it properly. Looks like you did :-)
But I have a few comments:
Review struct hrtimer_cpu_base *get_target_base · line 220
Quoted 11 lines
(quoted diff) +7
-0@@ -208,6 +211,13 @@ struct hrtimer_cpu_base *get_target_base if (static_branch_likely(&timers_migration_enabled) && !pinned) return &per_cpu(hrtimer_bases, get_nohz_timer_target()); #endif +#ifdef CONFIG_HOTPLUG_CPU + if (unlikely(!base->online)) { + int cpu = cpumask_any_and(cpu_online_mask, housekeeping_cpumask(HK_TYPE_TIMER)); + + return &per_cpu(hrtimer_bases, cpu); + } +#endif
This should be at the beginning of get_target_base(), otherwise the target may end up being the local one.
Review switch_hrtimer_base(struct hrtimer *time · line 270
Quoted 12 lines
(quoted diff) +1
-1return base; }@@ -254,7 +264,7 @@ switch_hrtimer_base(struct hrtimer *time raw_spin_unlock(&base->cpu_base->lock); raw_spin_lock(&new_base->cpu_base->lock); - if (new_cpu_base != this_cpu_base && + if (new_cpu_base != this_cpu_base && this_cpu_base->online && hrtimer_check_target(timer, new_base)) { raw_spin_unlock(&new_base->cpu_base->lock); raw_spin_lock(&base->cpu_base->lock);
This forget the case where the elected remote target is the same as the old one, but its next event is after the timer. In that case this default to local, even if it is offline.
How about this? (untested yet)
include/linux/hrtimer_defs.h +1 -0
@@ -125,6 +125,7 @@ struct hrtimer_cpu_base {ktime_t softirq_expires_next;struct hrtimer *softirq_next_timer;struct hrtimer_clock_base clock_base[HRTIMER_MAX_CLOCK_BASES ]; +call_single_data_t csd; }____cacheline_aligned ;
kernel/time/hrtimer.c +73 -18
172 lines not on this page · raw message
Re: [PATCH 1/3 v3] hrtimers: Force migrate away hrtimers queued after CPUHP_AP_HRTIMERS_DYING
Quoted 12 lines · Frederic Weisbecker
Quoted 5 lines · Thomas Gleixner
Review
Quoted 2 lines
(quoted diff) +2
-0+ if (enqueue_hrtimer(timer, new_base, mode)) + smp_call_function_single_async(cpu, &new_cpu_base->csd);Duh. This reimplementation of
switch_hrtimer_base()is really aweful. We can be smarter than that. Untested patch below.Indeed, and I tried to "fix"
switch_hrtimer_base()but didn't managed to do it properly. Looks like you did :-)But I have a few comments:
Quoted 14 lines
Review struct hrtimer_cpu_base *get_target_base · line 220
Quoted 11 lines
(quoted diff) +7
-0@@ -208,6 +211,13 @@ struct hrtimer_cpu_base *get_target_base if (static_branch_likely(&timers_migration_enabled) && !pinned) return &per_cpu(hrtimer_bases, get_nohz_timer_target()); #endif +#ifdef CONFIG_HOTPLUG_CPU + if (unlikely(!base->online)) { + int cpu = cpumask_any_and(cpu_online_mask, housekeeping_cpumask(HK_TYPE_TIMER)); + + return &per_cpu(hrtimer_bases, cpu); + } +#endifThis should be at the beginning of
get_target_base(), otherwise the target may end up being the local one.
Indeed. As I said, it's untested.
Quoted 9 lines
Quoted 5 lines
kernel/time/hrtimer.c +1
-1-if (new_cpu_base != this_cpu_base && +if (new_cpu_base != this_cpu_base && this_cpu_base->online &&hrtimer_check_target (timer, new_base)) {raw_spin_unlock (&new_base->cpu_base->lock);
raw_spin_lock(&base->cpu_base->lock);This forget the case where the elected remote target is the same as the old one, but its next event is after the timer. In that case this default to local, even if it is offline.
*blink*
Quoted 4 lines
How about this? (untested yet)
kernel/time/hrtimer.c +1
-2-static int -hrtimer_check_target (struct hrtimer *timer,struct hrtimer_clock_base *new_base) +static bool
kernel/time/hrtimer.c struct hrtimer_clock_base *lock_hrtimer_base(const struct hrtimer *timer, · line 220
Quoted 10 lines
kernel/time/hrtimer.c +7
-0+hrtimer_suitable_target(struct hrtimer *timer, + struct hrtimer_clock_base *new_base, + struct hrtimer_cpu_base *new_cpu_base, + struct hrtimer_cpu_base *this_cpu_base) { ktime_t expires; +/* The local CPU clockevent can be reprogrammed */ +if (new_cpu_base == this_cpu_base) +return true;
That needs a comment explaining that @new_cpu_base is guaranteed not to be @this_cpu_base if @this_cpu_base->online == false due to the magic in get_target_base(). That's far from obvious.
I had to stare at it 5 times to convince myself that this is correct.
Other than that, this looks about right.
Thanks,
tglx
Frederic Weisbecker
maintainerPATCH v3 2/3 rcu: Remove swake_up_one_online() bandaid
In reply to Frederic WeisbeckerIt's now ok to perform a wake-up from an offline CPU because the resulting armed scheduler bandwidth hrtimers are now correctly targeted by hrtimer infrastructure.
Remove the obsolete hackerry.
--- kernel/rcu/tree.c | 34 +--------------------------------- kernel/rcu/tree_exp.h | 2 +- kernel/rcu/tree_nocb.h | 2 +- 3 files changed, 3 insertions(+), 35 deletions(-)
kernel/rcu/tree.c +1 -33
@@ -1061,38 +1061,6 @@ static bool rcu_future_gp_cleanup(struct rcu_node *rnp)return needmore; } -static void swake_up_one_online_ipi (void *arg) -{ -struct swait_queue_head *wqh = arg; - -swake_up_one (wqh); -} - -static void swake_up_one_online (struct swait_queue_head *wqh) -{ -int cpu =get_cpu (); - -/* -* If called from rcutree_report_cpu_starting(), wake up -* is dangerous that late in the CPU-down hotplug process. The -* scheduler might queue an ignored hrtimer. Defer the wake up -* to an online CPU instead. -*/ -if (unlikely (cpu_is_offline (cpu))) { -int target; - - target =cpumask_any_and (housekeeping_cpumask (HK_TYPE_RCU ), - cpu_online_mask); - -smp_call_function_single (target, swake_up_one_online_ipi, - wqh,0 ); -put_cpu (); - }else { -put_cpu (); -swake_up_one (wqh); - } -} -/* * Awaken the grace-period kthread. Don't do a self-awaken (unless in an * interrupt or softirq handler, in which case we just might immediately
@@ -1117,7 +1085,7 @@ static void rcu_gp_kthread_wake(void)return ;WRITE_ONCE (rcu_state.gp_wake_time, jiffies);WRITE_ONCE (rcu_state.gp_wake_seq,READ_ONCE (rcu_state.gp_seq)); -swake_up_one_online (&rcu_state.gp_wq); +swake_up_one (&rcu_state.gp_wq); }/*
kernel/rcu/tree_exp.h +1 -1
@@ -200,7 +200,7 @@ static void __rcu_report_exp_rnp(struct rcu_node *rnp,if (rnp->parent ==NULL ) {raw_spin_unlock_irqrestore_rcu_node (rnp, flags);if (wake) -swake_up_one_online (&rcu_state.expedited_wq); +swake_up_one (&rcu_state.expedited_wq);break ; }
kernel/rcu/tree_nocb.h +1 -1
@@ -216,7 +216,7 @@ static bool __wake_nocb_gp(struct rcu_data *rdp_gp,raw_spin_unlock_irqrestore (&rdp_gp->nocb_gp_lock, flags);if (needwake) {trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("DoWake" )); -swake_up_one_online (&rdp_gp->nocb_gp_wq); +swake_up_one (&rdp_gp->nocb_gp_wq); }return needwake;
Signature · 1 line
2.46.0
Frederic Weisbecker
maintainerPATCH v3 3/3 Revert "rcu/nocb: Fix rcuog wake-up from offline softirq"
In reply to Frederic WeisbeckerThis reverts commit f7345ccc62a4b880cf76458db5f320725f28e400.
swake_up_one_online() has been removed because hrtimers can now assign a proper online target to hrtimers queued from offline CPUs. Therefore remove the related hackery.
--- kernel/rcu/tree_nocb.h | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
kernel/rcu/tree_nocb.h +1 -7
@@ -554,19 +554,13 @@ static void __call_rcu_nocb_wake(struct rcu_data *rdp, bool was_alldone,rcu_nocb_unlock (rdp);wake_nocb_gp_defer (rdp,RCU_NOCB_WAKE_LAZY ,TPS ("WakeLazy" )); - }else if (!irqs_disabled_flags (flags) &&cpu_online (rdp->cpu)) { + }else if (!irqs_disabled_flags (flags)) {/* ... if queue was empty ... */ rcu_nocb_unlock (rdp);wake_nocb_gp (rdp, false);trace_rcu_nocb_wake (rcu_state.name, rdp->cpu,TPS ("WakeEmpty" )); }else { -/* -* Don't do the wake-up upfront on fragile paths. -* Also offline CPUs can't call swake_up_one_online() from -* (soft-)IRQs. Rely on the final deferred wake-up from -* rcutree_report_cpu_dead() -*/ rcu_nocb_unlock (rdp);wake_nocb_gp_defer (rdp,RCU_NOCB_WAKE ,TPS ("WakeEmptyIsDeferred" ));
Signature · 1 line
2.46.0