kthread/smpboot: More fixes...
Subsystems (2)
Peter Zijlstra
maintainercover kthread/smpboot: More fixes...
These patches were tested with hotplug and /proc/sys/kernel/watchdog_cpumask manipulations.
I used 'grep watchdog_fn /proc/timer_list | wc -l' to verify the expected number of active timers.
Please have a look..
--- include/linux/cpuhotplug.h | 1 + include/linux/kthread.h | 1 - include/linux/nmi.h | 5 ++ include/linux/sched.h | 2 +- include/linux/smpboot.h | 15 +----- kernel/cpu.c | 5 ++ kernel/kthread.c | 34 +++++++++--- kernel/sched/core.c | 31 ++++------- kernel/smpboot.c | 54 ++----------------- kernel/watchdog.c | 132 +++++++++++++++++++-------------------------- 10 files changed, 112 insertions(+), 168 deletions(-)
Peter Zijlstra
maintainerPATCH 1/4 kthread, sched: Fix kthread_parkme() (again...)
Gaurav reports that commit:
85f1abe0019f ("kthread, sched/wait: Fix kthread_parkme() completion issue")
isn't working for him. Because of the following race:
Quoted 25 lines
controller Thread CPUHP Thread
takedown_cpukthread_parkkthread_parkme
SetKTHREAD_SHOULD_PARK
smpboot_thread_fnset Task interruptible
wake_up_process
if (!(p->state & state))goto out; Kthread_parkme SET TASK_PARKED schedule raw_spin_lock(&rq->lock)
ttwu_remote
waiting for__task_rq_lock
context_switch
finish_lock_switch
Quoted 5 lines
Case
TASK_PARKEDkthread_park_completeSET Running
Furthermore, Oleg noticed that the whole scheduler TASK_PARKED handling is buggered because the TASK_DEAD thing is done with preemption disabled, the current code can still complete early on preemption :/
So basically revert that earlier fix and go with a variant of the alternative mentioned in the commit. Promote TASK_PARKED to special state to avoid the store-store issue on task->state leading to the WARN in kthread_unpark() -> __kthread_bind().
But in addition, add wait_task_inactive() to kthread_park() to ensure the task really is PARKED when we return from kthread_park(). This avoids the whole kthread still gets migrated nonsense -- although it would be really good to get this done differently.
--- include/linux/kthread.h | 1 - include/linux/sched.h | 2 +- kernel/kthread.c | 30 ++++++++++++++++++++++++------ kernel/sched/core.c | 31 +++++++++++-------------------- 4 files changed, 36 insertions(+), 28 deletions(-)
include/linux/kthread.h +0 -1
@@ -62,7 +62,6 @@ void *kthread_probe_data(struct task_strint kthread_park (struct task_struct *k);void kthread_unpark (struct task_struct *k);void kthread_parkme (void ); -void kthread_park_complete (struct task_struct *k);int kthreadd (void *unused);extern struct task_struct *kthreadd_task;
include/linux/sched.h +1 -1
@@ -117,7 +117,7 @@ struct task_group;* the comment with set_special_state(). */ #define is_special_task_state (state) \ - ((state) & (__TASK_STOPPED | __TASK_TRACED | TASK_DEAD)) + ((state) & (__TASK_STOPPED | __TASK_TRACED | TASK_PARKED | TASK_DEAD))#define __set_current_state (state_value) \ do { \
kernel/kthread.c +24 -6
@@ -177,9 +177,20 @@ void *kthread_probe_data(struct task_strstatic void __kthread_parkme (struct kthread *self) {for (;;) { -set_current_state (TASK_PARKED ); +/* +* TASK_PARKED is a special state; we must serialize against +* possible pending wakeups to avoid store-store collisions on +* task->state. +* +* Such a collision might possibly result in the task state +* changin from TASK_PARKED and us failing the +* wait_task_inactive() in kthread_park(). +*/ +set_special_state (TASK_PARKED );if (!test_bit (KTHREAD_SHOULD_PARK , &self->flags))break ; + +complete_all (&self->parked);schedule (); }__set_current_state (TASK_RUNNING );
@@ -191,11 +202,6 @@ void kthread_parkme(void) }EXPORT_SYMBOL_GPL (kthread_parkme); -void kthread_park_complete (struct task_struct *k) -{ -complete_all (&to_kthread (k)->parked); -} -static int kthread (void *_create) {/* Copy data: it's on kthread's stack */
@@ -461,6 +467,9 @@ void kthread_unpark(struct task_struct *reinit_completion (&kthread->parked);clear_bit (KTHREAD_SHOULD_PARK , &kthread->flags); +/* +* __kthread_parkme() will either see !SHOULD_PARK or get the wakeup. +*/ wake_up_state (k,TASK_PARKED ); }EXPORT_SYMBOL_GPL (kthread_unpark);
@@ -487,7 +496,16 @@ int kthread_park(struct task_struct *k)set_bit (KTHREAD_SHOULD_PARK , &kthread->flags);if (k != current) {wake_up_process (k); +/* +* Wait for __kthread_parkme() to complete(), this means we +* _will_ have TASK_PARKED and are about to call schedule(). +*/ wait_for_completion (&kthread->parked); +/* +* Now wait for that schedule() to complete and the task to +* get scheduled out. +*/ +WARN_ON_ONCE (!wait_task_inactive (k,TASK_PARKED )); }return 0 ;
kernel/sched/core.c +11 -20
@@ -7,7 +7,6 @@*/ #include "sched.h" -#include <linux/kthread.h> #include <linux/nospec.h> #include <asm/switch_to.h>
@@ -2701,28 +2700,20 @@ static struct rq *finish_task_switch(strmembarrier_mm_sync_core_before_usermode (mm);mmdrop (mm); } -if (unlikely (prev_state & (TASK_DEAD |TASK_PARKED ))) { -switch (prev_state) { -case TASK_DEAD : -if (prev->sched_class->task_dead) - prev->sched_class->task_dead (prev); - -/* -* Remove function-return probe instances associated with this -* task and put them back on the free list. -*/ -kprobe_flush_task (prev); +if (unlikely (prev_state ==TASK_DEAD )) { +if (prev->sched_class->task_dead) + prev->sched_class->task_dead (prev); -/* Task is done with its stack. */ -put_task_stack (prev); +/* +* Remove function-return probe instances associated with this +* task and put them back on the free list. +*/ +kprobe_flush_task (prev); -put_task_struct (prev); -break ; +/* Task is done with its stack. */ +put_task_stack (prev); -case TASK_PARKED : -kthread_park_complete (prev); -break ; - } +put_task_struct (prev); }tick_nohz_task_switch ();
Peter Zijlstra
maintainerPATCH 2/4 watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
In reply to Peter ZijlstraOleg suggested to replace the "watchdog/%u" threads with cpu_stop_work. That removes one thread per cpu while at the same time fixes softlockup vs SCHED_DEADLINE.
But more importantly, it does away with the single smpboot_update_cpumask_percpu_thread() user, which allows cleanups/shrinkage of the smpboot interface.
--- include/linux/cpuhotplug.h | 1 include/linux/nmi.h | 5 + kernel/cpu.c | 5 + kernel/watchdog.c | 132 +++++++++++++++++++-------------------------- 4 files changed, 67 insertions(+), 76 deletions(-)
include/linux/cpuhotplug.h +1 -0
@@ -164,6 +164,7 @@ enum cpuhp_state {CPUHP_AP_PERF_POWERPC_NEST_IMC_ONLINE ,CPUHP_AP_PERF_POWERPC_CORE_IMC_ONLINE ,CPUHP_AP_PERF_POWERPC_THREAD_IMC_ONLINE , +CPUHP_AP_WATCHDOG_ONLINE ,CPUHP_AP_WORKQUEUE_ONLINE ,CPUHP_AP_RCUTREE_ONLINE , CPUHP_AP_ONLINE_DYN,
include/linux/nmi.h +5 -0
@@ -33,10 +33,15 @@ extern int sysctl_hardlockup_all_cpu_bac#define sysctl_hardlockup_all_cpu_backtrace 0#endif /* !CONFIG_SMP */ +extern int lockup_detector_online_cpu (unsigned int cpu); +extern int lockup_detector_offline_cpu (unsigned int cpu); +#else /* CONFIG_LOCKUP_DETECTOR */ static inline void lockup_detector_init (void ) { }static inline void lockup_detector_soft_poweroff (void ) { }static inline void lockup_detector_cleanup (void ) { } +#define lockup_detector_online_cpu NULL +#define lockup_detector_offline_cpu NULL#endif /* !CONFIG_LOCKUP_DETECTOR */ #ifdef CONFIG_SOFTLOCKUP_DETECTOR
kernel/cpu.c +5 -0
@@ -1344,6 +1344,11 @@ static struct cpuhp_step cpuhp_hp_states .startup.single = perf_event_init_cpu, .teardown.single = perf_event_exit_cpu, }, + [CPUHP_AP_WATCHDOG_ONLINE] = { + .name = "lockup_detector:online", + .startup.single = lockup_detector_online_cpu, + .teardown.single = lockup_detector_offline_cpu, + }, [CPUHP_AP_WORKQUEUE_ONLINE] = { .name = "workqueue:online", .startup.single = workqueue_online_cpu,
kernel/watchdog.c +56 -76
241 lines not on this page · raw message
Peter Zijlstra
maintainerRe: [PATCH 2/4] watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
Quoted 5 lines · Peter Zijlstra
kernel/watchdog.c +3
-0+static int softlockup_stop_fn (void *data) { +watchdog_disable (smp_processor_id ()); +return 0 ; }
kernel/watchdog.c static void watchdog_disable(unsigned in · line 518
Quoted 10 lines
kernel/watchdog.c +7
-0+static void softlockup_stop_all (void ) { +int cpu; + +for_each_cpu (cpu, &watchdog_allowed_mask) +stop_one_cpu (cpu, softlockup_stop_fn,NULL ); + +cpumask_clear (&watchdog_allowed_mask); }
Bugger, that one doesn't quite work.. watchdog_disable() ends up calling a sleeping function. I forgot to enable all the debug cruft when testing..
Let me try and fix that..
Peter Zijlstra
maintainerRe: [PATCH 2/4] watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
Quoted 22 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
(quoted diff) +3
-0+static int softlockup_stop_fn(void *data) { + watchdog_disable(smp_processor_id()); + return 0; }Review
Quoted 10 lines
(quoted diff) +7
-0+static void softlockup_stop_all(void) { + int cpu; + + for_each_cpu(cpu, &watchdog_allowed_mask) + stop_one_cpu(cpu, softlockup_stop_fn, NULL); + + cpumask_clear(&watchdog_allowed_mask); }Bugger, that one doesn't quite work..
watchdog_disable()ends up calling a sleeping function. I forgot to enable all the debug cruft when testing..
Quoted 1 line
Let me try and fix that..
The below seems to fix that, and now I can complete all the tests without triggering debug nonsense ;-)
kernel/watchdog.c +2 -2
@@ -512,7 +512,7 @@ static void softlockup_stop_all(void)int cpu;for_each_cpu (cpu, &watchdog_allowed_mask) -stop_one_cpu (cpu, softlockup_stop_fn,NULL ); +smp_call_on_cpu (cpu, softlockup_stop_fn,NULL , false);cpumask_clear (&watchdog_allowed_mask); }
@@ -529,7 +529,7 @@ static void softlockup_start_all(void)cpumask_copy (&watchdog_allowed_mask, &watchdog_cpumask);for_each_cpu (cpu, &watchdog_allowed_mask) -stop_one_cpu (cpu, softlockup_start_fn,NULL ); +smp_call_on_cpu (cpu, softlockup_start_fn,NULL , false); }int lockup_detector_online_cpu (unsigned int cpu)
Re: [PATCH 2/4] watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
In reply to Peter ZijlstraQuoted 9 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra
(quoted diff) +3
-0+static int softlockup_stop_fn(void *data) { + watchdog_disable(smp_processor_id()); + return 0; }
Quoted 13 lines
Review
Quoted 9 lines
(quoted diff) +7
-0+static void softlockup_stop_all(void) { + int cpu; + + for_each_cpu(cpu, &watchdog_allowed_mask) + stop_one_cpu(cpu, softlockup_stop_fn, NULL); + + cpumask_clear(&watchdog_allowed_mask); }Bugger, that one doesn't quite work..
watchdog_disable()ends up calling a sleeping function. I forgot to enable all the debug cruft when testing..
And probably there is another problem. Both watchdog_disable(cpu) and watchdog_nmi_disable(cpu) assume that cpu == smp_processor_id(), this arg is simply ignored.
but lockup_detector_offline_cpu(cpu) is called by cpuhp_invoke_callback(), so in this case watchdog_disable(dying_cpu) is simply wrong.
May be we can do something like below? Then softlockup_stop_all() can simply do
for_each_cpu(cpu, &watchdog_allowed_mask) watchdog_disable(cpu);
watchdog_nmi_disable() is __weak, but at first glance arch/sparc/kernel/nmi.c does everything correctly.
Oleg.
kernel/watchdog.c +3 -3
@@ -108,13 +108,13 @@ __setup("hardlockup_all_cpu_backtrace=",*/ int __weak watchdog_nmi_enable (unsigned int cpu) { -hardlockup_detector_perf_enable (); +hardlockup_detector_perf_enable (cpu);return 0 ; }void __weak watchdog_nmi_disable (unsigned int cpu) { -hardlockup_detector_perf_disable (); +hardlockup_detector_perf_disable (cpu); }/* Return 0, if a NMI watchdog is available. Error code otherwise */
@@ -479,7 +479,7 @@ static void watchdog_enable(unsigned intstatic void watchdog_disable (unsigned int cpu) { -struct hrtimer *hrtimer =this_cpu_ptr (&watchdog_hrtimer); +struct hrtimer *hrtimer =per_cpu_ptr (&watchdog_hrtimer, cpu);watchdog_set_prio (SCHED_NORMAL ,0 );/*
kernel/watchdog_hld.c +10 -11
@@ -162,9 +162,8 @@ static void watchdog_overflow_callback(sreturn ; } -static int hardlockup_detector_event_create (void ) +static int hardlockup_detector_event_create (int cpu) { -unsigned int cpu =smp_processor_id ();struct perf_event_attr *wd_attr;struct perf_event *evt;
@@ -179,37 +178,37 @@ static int hardlockup_detector_event_cre PTR_ERR(evt));return PTR_ERR (evt); } -this_cpu_write (watchdog_ev, evt); +raw_cpu_write (watchdog_ev, evt);return 0 ; }/** * hardlockup_detector_perf_enable - Enable the local event */ -void hardlockup_detector_perf_enable (void ) +void hardlockup_detector_perf_enable (int cpu) { -if (hardlockup_detector_event_create ()) +if (hardlockup_detector_event_create (cpu))return ;/* use original value for check */ if (!atomic_fetch_inc (&watchdog_cpus))pr_info ("Enabled. Permanently consumes one hw-PMU counter.\n" ); -perf_event_enable (this_cpu_read (watchdog_ev)); +perf_event_enable (raw_cpu_read (watchdog_ev)); }/** * hardlockup_detector_perf_disable - Disable the local event */ -void hardlockup_detector_perf_disable (void ) +void hardlockup_detector_perf_disable (int cpu) { -struct perf_event *event =this_cpu_read (watchdog_ev); +struct perf_event *event =per_cpu_read (watchdog_ev, cpu);if (event) {perf_event_disable (event); -this_cpu_write (watchdog_ev,NULL ); -this_cpu_write (dead_event, event); -cpumask_set_cpu (smp_processor_id (), &dead_events_mask); +raw_cpu_write (watchdog_ev, cpu,NULL ); +raw_cpu_write (dead_event, cpu, event); +cpumask_set_cpu (cpu, &dead_events_mask);atomic_dec (&watchdog_cpus); } }
Peter Zijlstra
maintainerRe: [PATCH 2/4] watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work
Quoted 6 lines · Oleg Nesterov
And probably there is another problem. Both
watchdog_disable(cpu)andwatchdog_nmi_disable(cpu)assume that cpu ==smp_processor_id(), this arg is simply ignored.but
lockup_detector_offline_cpu(cpu)is called bycpuhp_invoke_callback(), so in this casewatchdog_disable(dying_cpu)is simply wrong.
But at this point, the cpuhp_invoke_callback() is ran from the dying CPU still, so dying_cpu == this_cpu. I actually have a WARN in both watchdog_{dis,en}able() to verify this assumption.
Quoted 7 lines
May be we can do something like below? Then
softlockup_stop_all()can simply dofor_each_cpu(cpu, &watchdog_allowed_mask) watchdog_disable(cpu);
watchdog_nmi_disable()is__weak, but at first glancearch/sparc/kernel/nmi.cdoes everything correctly.
I prefer to not do that and keep the current assumption. While it would work for the disable, it the above form will not work for enable (we really must start hrtimers on the right CPU) and that would bring some asymmetry.
Peter Zijlstra
maintainerPATCH 3/4 smpboot: Remove cpumask from the API
In reply to Peter ZijlstraNow that the sole use of the whole smpboot_*cpumask() API is gone, remove it.
--- include/linux/smpboot.h | 15 ------------- kernel/smpboot.c | 54 ++++-------------------------------------------- 2 files changed, 6 insertions(+), 63 deletions(-)
include/linux/smpboot.h +1 -14
@@ -25,8 +25,6 @@ struct smpboot_thread_data;* parked (cpu offline) * @unpark: Optional unpark function, called when the thread is * unparked (cpu online) -* @cpumask: Internal state. To update which threads are unparked, -* call smpboot_update_cpumask_percpu_thread(). * @selfparking: Thread is not parked by the park function. * @thread_comm: The base name of the thread */
@@ -40,23 +38,12 @@ struct smp_hotplug_thread {void (*cleanup)(unsigned int cpu,bool online);void (*park)(unsigned int cpu);void (*unpark)(unsigned int cpu); -cpumask_var_t cpumask;bool selfparking; const char *thread_comm; }; -int smpboot_register_percpu_thread_cpumask (struct smp_hotplug_thread *plug_thread, -const struct cpumask *cpumask); - -static inline int -smpboot_register_percpu_thread (struct smp_hotplug_thread *plug_thread) -{ -return smpboot_register_percpu_thread_cpumask (plug_thread, - cpu_possible_mask); -} +int smpboot_register_percpu_thread (struct smp_hotplug_thread *plug_thread);void smpboot_unregister_percpu_thread (struct smp_hotplug_thread *plug_thread); -void smpboot_update_cpumask_percpu_thread (struct smp_hotplug_thread *plug_thread, -const struct cpumask *);#endif
kernel/smpboot.c +5 -49
98 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH 4/4 kthread: Simplify kthread_park() completion
In reply to Peter ZijlstraNow that smpboot_update_cpumask_percpu_thread() is gone, we no longer have anybody calling kthread_park() on already parked threads. So revert commit:
b1f5b378e126 ("kthread: Allow kthread_park() on a parked kthread")
--- kernel/kthread.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/kthread.c +4 -2
@@ -190,7 +190,7 @@ static void __kthread_parkme(struct kthrif (!test_bit (KTHREAD_SHOULD_PARK , &self->flags))break ; -complete_all (&self->parked); +complete (&self->parked);schedule (); }__set_current_state (TASK_RUNNING );
@@ -465,7 +465,6 @@ void kthread_unpark(struct task_struct *if (test_bit (KTHREAD_IS_PER_CPU , &kthread->flags))__kthread_bind (k, kthread->cpu,TASK_PARKED ); -reinit_completion (&kthread->parked);clear_bit (KTHREAD_SHOULD_PARK , &kthread->flags);/* * __kthread_parkme() will either see !SHOULD_PARK or get the wakeup.
@@ -493,6 +492,9 @@ int kthread_park(struct task_struct *k)if (WARN_ON (k->flags &PF_EXITING ))return -ENOSYS ; +if (WARN_ON_ONCE (test_bit (KTHREAD_SHOULD_PARK , &kthread->flags))) +return -EBUSY ; +set_bit (KTHREAD_SHOULD_PARK , &kthread->flags);if (k != current) {wake_up_process (k);
Re: [PATCH 4/4] kthread: Simplify kthread_park() completion
Peter,
I am travelling till the end of the next week, unlikely I will be able to reply to emails or even read them.
But I want very much to comment this change,
Quoted 6 lines · Peter Zijlstra
Now that
smpboot_update_cpumask_percpu_thread()is gone, we no longer have anybody callingkthread_park()on already parked threads. So revert commit:
b1f5b378e126("kthread: Allowkthread_park()on a parked kthread")
Great, I obviously like this patch but the changelog should be fixed ;)
smpboot_update_cpumask_percpu_thread() was actually fine. And we can (should) revert this commit in any case. Unless I am totally confused.
So how this code
for_each_cpu_and(cpu, &tmp, cpu_online_mask) smpboot_park_thread(plug_thread, cpu);
in smpboot_update_cpumask_percpu_thread() can hit a KTHREAD_SHOULD_PARK thread? Lets look into kernel test robot's .config:
CONFIG_NR_CPUS=1
Now look at NR_CPUS==1 version of for_each_cpu* helpers:
#define for_each_cpu(cpu, mask) \ for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask) #define for_each_cpu_not(cpu, mask) \ for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask) #define for_each_cpu_wrap(cpu, mask, start) \ for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask, (void)(start)) #define for_each_cpu_and(cpu, mask, and) \ for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask, (void)and)
See? They all ignore the "mask" argument, and this is obviously wrong.
So even if the "tmp" cpumask is empty the code above always does
smpboot_park_thread(plug_thread, 0);
and hits the already parked kthread.
Oleg.
Peter Zijlstra
maintainerRe: [PATCH 4/4] kthread: Simplify kthread_park() completion
Quoted 9 lines · Oleg Nesterov
in
smpboot_update_cpumask_percpu_thread()can hit aKTHREAD_SHOULD_PARKthread? Lets look into kernel test robot's .config:
CONFIG_NR_CPUS=1Now look at NR_CPUS==1 version of for_each_cpu* helpers:
#define for_each_cpu(cpu, mask) \ for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask)
Argh, that issue again.
Quoted 5 lines
So even if the "tmp" cpumask is empty the code above always does
smpboot_park_thread(plug_thread, 0);and hits the already parked kthread.
OK, I'll write a new Changelog.
Peter Zijlstra
maintainerRe: [PATCH 4/4] kthread: Simplify kthread_park() completion
In reply to Oleg NesterovHi Oleg,
Quoted 2 lines · Oleg Nesterov
I am travelling till the end of the next week, unlikely I will be able to reply to emails or even read them.
Should I post a new version; with updated Changelogs for you?
Re: [PATCH 4/4] kthread: Simplify kthread_park() completion
Hi Peter,
Quoted 8 lines · Peter Zijlstra
Hi Oleg,
Quoted 2 lines · Oleg Nesterov
I am travelling till the end of the next week, unlikely I will be able to reply to emails or even read them.
Should I post a new version; with updated Changelogs for you?
IIRC you have already sent v2 with the updated changelog which explains the problem with for_each_cpu? That series looked good to me, thanks!
Oleg.