sched: Fix hot-unplug regressions
Subsystems (3)
Peter Zijlstra
maintainercover sched: Fix hot-unplug regressions
Hi,
These patches (no longer 4), seems to fix all the hotplug regressions as per nearly a 100 18*SRCU-P runs over-night.
I did clean up the patches, so possibly I wrecked it again. I've started new runs and will again leave them running over-night.
Paul, if you could please also throw your monster machine at it.
Peter Zijlstra
maintainerPATCH 1/8 sched/core: Print out straggler tasks in sched_cpu_dying()
From: Valentin Schneider <[email protected]>
Since commit
1cf12e08bc4d ("sched/hotplug: Consolidate task migration on CPU unplug")
tasks are expected to move themselves out of a out-going CPU. For most tasks this will be done automagically via BALANCE_PUSH, but percpu kthreads will have to cooperate and move themselves away one way or another.
Currently, some percpu kthreads (workqueues being a notable exemple) do not cooperate nicely and can end up on an out-going CPU at the time sched_cpu_dying() is invoked.
Print the dying rq's tasks to shed some light on the stragglers.
--- kernel/sched/core.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-)
kernel/sched/core.c +23 -1
@@ -7580,6 +7580,25 @@ static void calc_load_migrate(struct rqatomic_long_add (delta, &calc_load_tasks); } +static void dump_rq_tasks (struct rq *rq,const char *loglvl) +{ +struct task_struct *g, *p; +int cpu =cpu_of (rq); + +lockdep_assert_held (&rq->lock); + +printk ("%sCPU%d enqueued tasks (%u total):\n" , loglvl, cpu, rq->nr_running); +for_each_process_thread (g, p) { +if (task_cpu (p) != cpu) +continue ; + +if (!task_on_rq_queued (p)) +continue ; + +printk ("%s\tpid: %d, name: %s\n" , loglvl, p->pid, p->comm); + } +} +int sched_cpu_dying (unsigned int cpu) {struct rq *rq =cpu_rq (cpu);
@@ -7589,7 +7608,10 @@ int sched_cpu_dying(unsigned int cpu)sched_tick_stop (cpu);rq_lock_irqsave (rq, &rf); -BUG_ON (rq->nr_running !=1 ||rq_has_pinned_tasks (rq)); +if (rq->nr_running !=1 ||rq_has_pinned_tasks (rq)) { +WARN (true,"Dying CPU not properly vacated!" ); +dump_rq_tasks (rq,KERN_WARNING ); + }rq_unlock_irqrestore (rq, &rf);calc_load_migrate (rq);
Peter Zijlstra
maintainerPATCH 2/8 workqueue: Use cpu_possible_mask instead of cpu_active_mask to break affinity
In reply to Peter ZijlstraFrom: Lai Jiangshan <[email protected]>
The scheduler won't break affinity for us any more, and we should "emulate" the same behavior when the scheduler breaks affinity for us. The behavior is "changing the cpumask to cpu_possible_mask".
And there might be some other CPUs online later while the worker is still running with the pending work items. The worker should be allowed to use the later online CPUs as before and process the work items ASAP. If we use cpu_active_mask here, we can't achieve this goal but using cpu_possible_mask can.
--- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/workqueue.c +1 -1
@@ -4920,7 +4920,7 @@ static void unbind_workers(int cpu)raw_spin_unlock_irq (&pool->lock);for_each_pool_worker (worker, pool) -WARN_ON_ONCE (set_cpus_allowed_ptr (worker->task, cpu_active_mask) <0 ); +WARN_ON_ONCE (set_cpus_allowed_ptr (worker->task, cpu_possible_mask) <0 );mutex_unlock (&wq_pool_attach_mutex);
Peter Zijlstra
maintainerPATCH 3/8 sched: Dont run cpu-online with balance_push() enabled
In reply to Peter ZijlstraWe don't need to push away tasks when we come online, mark the push complete right before the CPU dies.
XXX hotplug state machine has trouble with rollback here.
--- kernel/sched/core.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
kernel/sched/core.c +14 -2
@@ -7320,10 +7320,12 @@ static void balance_push_set(int cpu, bostruct rq_flags rf;rq_lock_irqsave (rq, &rf); -if (on) +if (on) { +WARN_ON_ONCE (rq->balance_callback); rq->balance_callback = &balance_push_callback; -else + }else if (rq->balance_callback == &balance_push_callback) { rq->balance_callback =NULL ; + }rq_unlock_irqrestore (rq, &rf); }
@@ -7441,6 +7443,10 @@ int sched_cpu_activate(unsigned int cpu)struct rq *rq =cpu_rq (cpu);struct rq_flags rf; +/* +* Make sure that when the hotplug state machine does a roll-back +* we clear balance_push. Ideally that would happen earlier... +*/ balance_push_set (cpu, false);#ifdef CONFIG_SCHED_SMT
@@ -7608,6 +7614,12 @@ int sched_cpu_dying(unsigned int cpu) }rq_unlock_irqrestore (rq, &rf); +/* +* Should really be after we clear cpu_online(), but we're in +* stop_machine(), so it all works. +*/ +balance_push_set (cpu, false); +calc_load_migrate (rq);update_max_interval ();nohz_balance_exit_idle (rq);
Peter Zijlstra
maintainerRe: [PATCH 3/8] sched: Dont run cpu-online with balance_push() enabled
kernel/sched/core.c int sched_cpu_dying(unsigned int cpu) · line 7621
Quoted 9 lines · Peter Zijlstra
kernel/sched/core.c +5
-0@@ -7608,6 +7614,12 @@ int sched_cpu_dying(unsigned int cpu) }rq_unlock_irqrestore (rq, &rf); +/* +* Should really be after we clear cpu_online(), but we're in +* stop_machine(), so it all works. +*/ +balance_push_set (cpu, false);
Looking at the RCU thing just now made me realize we run all the DYING notifiers with cpu_online() already false, so the above comment is wrong and ordering in fact perfect.
Peter Zijlstra
maintainerPATCH 4/8 kthread: Extract KTHREAD_IS_PER_CPU
In reply to Peter ZijlstraThere is a need to distinguish geniune per-cpu kthreads from kthreads that happen to have a single CPU affinity.
Geniune per-cpu kthreads are kthreads that are CPU affine for correctness, these will obviously have PF_KTHREAD set, but must also have PF_NO_SETAFFINITY set, lest userspace modify their affinity and ruins things.
However, these two things are not sufficient, PF_NO_SETAFFINITY is also set on other tasks that have their affinities controlled through other means, like for instance workqueues.
Therefore another bit is needed; it turns out kthread_create_per_cpu() already has such a bit: KTHREAD_IS_PER_CPU, which is used to make kthread_park()/kthread_unpark() work correctly.
Expose this flag and remove the implicit setting of it from kthread_create_on_cpu(); the io_uring usage of it seems dubious at best.
--- include/linux/kthread.h | 3 +++ kernel/kthread.c | 30 ++++++++++++++++++++++++++++-- kernel/smpboot.c | 1 + 3 files changed, 32 insertions(+), 2 deletions(-)
include/linux/kthread.h +3 -0
@@ -33,6 +33,9 @@ struct task_struct *kthread_create_on_cpunsigned int cpu, const char *namefmt); +void kthread_set_per_cpu (struct task_struct *k,int cpu); +bool kthread_is_per_cpu (struct task_struct *k); +/** * kthread_run - create and wake a thread. * @threadfn: the function to run until signal_pending(current).
kernel/kthread.c +26 -1
@@ -493,11 +494,36 @@ struct task_struct *kthread_create_on_cpreturn p;kthread_bind (p, cpu);/* CPU hotplug need to bind once again when unparking the thread. */ -set_bit (KTHREAD_IS_PER_CPU , &to_kthread (p)->flags);to_kthread (p)->cpu = cpu;return p; } +void kthread_set_per_cpu (struct task_struct *k,int cpu) +{ +struct kthread *kthread =to_kthread (k); +if (!kthread) +return ; + +WARN_ON_ONCE (!(k->flags &PF_NO_SETAFFINITY )); + +if (cpu <0 ) { +clear_bit (KTHREAD_IS_PER_CPU , &kthread->flags); +return ; + } + + kthread->cpu = cpu; +set_bit (KTHREAD_IS_PER_CPU , &kthread->flags); +} + +bool kthread_is_per_cpu (struct task_struct *k) +{ +struct kthread *kthread =to_kthread (k); +if (!kthread) +return false; + +return test_bit (KTHREAD_IS_PER_CPU , &kthread->flags); +} +/** * kthread_unpark - unpark a thread created by kthread_create(). * @k: thread created by kthread_create().
kernel/smpboot.c +1 -0
@@ -188,6 +188,7 @@ __smpboot_create_thread(struct smp_hotplkfree (td);return PTR_ERR (tsk); } +kthread_set_per_cpu (tsk, cpu);/* * Park the thread so that it could start right on the CPU * when it is available.
Peter Zijlstra
maintainerPATCH 5/8 workqueue: Tag bound workers with KTHREAD_IS_PER_CPU
In reply to Peter ZijlstraMark the per-cpu workqueue workers as KTHREAD_IS_PER_CPU.
Workqueues have unfortunate semantics in that per-cpu workers are not default flushed and parked during hotplug, however a subset does manual flush on hotplug and hard relies on them for correctness.
Therefore play silly games..
--- kernel/workqueue.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-)
kernel/workqueue.c +18 -7
@@ -1861,6 +1861,8 @@ static void worker_attach_to_pool(struct*/ if (pool->flags &POOL_DISASSOCIATED ) worker->flags |=WORKER_UNBOUND ; +else +kthread_set_per_cpu (worker->task, pool->cpu);list_add_tail (&worker->node, &pool->workers); worker->pool = pool;
@@ -1883,6 +1885,7 @@ static void worker_detach_from_pool(strumutex_lock (&wq_pool_attach_mutex); +kthread_set_per_cpu (worker->task,-1 );list_del (&worker->node); worker->pool =NULL ;
@@ -2368,6 +2371,7 @@ static int worker_thread(void *__worker)/* tell the scheduler that this is a workqueue worker */ set_pf_worker (true); woke_up: +kthread_parkme ();raw_spin_lock_irq (&pool->lock);/* am I supposed to die? */
@@ -2425,7 +2429,7 @@ static int worker_thread(void *__worker)move_linked_works (work, &worker->scheduled,NULL );process_scheduled_works (worker); } - }while (keep_working (pool)); + }while (keep_working (pool) && !kthread_should_park ());worker_set_flags (worker,WORKER_PREP ); sleep:
@@ -2437,9 +2441,12 @@ static int worker_thread(void *__worker)* event. */ worker_enter_idle (worker); -__set_current_state (TASK_IDLE ); +set_current_state (TASK_IDLE );raw_spin_unlock_irq (&pool->lock); -schedule (); + +if (!kthread_should_park ()) +schedule (); + goto woke_up; }
@@ -4919,8 +4926,10 @@ static void unbind_workers(int cpu)raw_spin_unlock_irq (&pool->lock); -for_each_pool_worker (worker, pool) +for_each_pool_worker (worker, pool) { +kthread_set_per_cpu (worker->task,-1 );WARN_ON_ONCE (set_cpus_allowed_ptr (worker->task, cpu_possible_mask) <0 ); + }mutex_unlock (&wq_pool_attach_mutex);
@@ -4972,9 +4981,11 @@ static void rebind_workers(struct worker* of all workers first and then clear UNBOUND. As we're called * from CPU_ONLINE, the following shouldn't fail. */ -for_each_pool_worker (worker, pool) -WARN_ON_ONCE (set_cpus_allowed_ptr (worker->task, - pool->attrs->cpumask) <0 ); +for_each_pool_worker (worker, pool) { +WARN_ON_ONCE (kthread_park (worker->task) <0 ); +kthread_set_per_cpu (worker->task, pool->cpu); +kthread_unpark (worker->task); + }raw_spin_lock_irq (&pool->lock);
Peter Zijlstra
maintainerPATCH 6/8 workqueue: Restrict affinity change to rescuer
In reply to Peter Zijlstracreate_worker() will already set the right affinity using kthread_bind_mask(), this means only the rescuer will need to change it's affinity.
Howveer, while in cpu-hot-unplug a regular task is not allowed to run on online&&!active as it would be pushed away quite agressively. We need KTHREAD_IS_PER_CPU to survive in that environment.
Therefore set the affinity after getting that magic flag.
--- kernel/workqueue.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
kernel/workqueue.c +3 -6
@@ -1849,12 +1849,6 @@ static void worker_attach_to_pool(structmutex_lock (&wq_pool_attach_mutex);/* -* set_cpus_allowed_ptr() will fail if the cpumask doesn't have any -* online CPUs. It'll be re-applied when any of the CPUs come up. -*/ -set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); - -/* * The wq_pool_attach_mutex ensures %POOL_DISASSOCIATED remains * stable across this function. See the comments above the flag * definition for details.
@@ -1864,6 +1858,9 @@ static void worker_attach_to_pool(structelse kthread_set_per_cpu (worker->task, pool->cpu); +if (worker->rescue_wq) +set_cpus_allowed_ptr (worker->task, pool->attrs->cpumask); +list_add_tail (&worker->node, &pool->workers); worker->pool = pool;
Peter Zijlstra
maintainerPATCH 7/8 sched: Fix CPU hotplug / tighten is_per_cpu_kthread()
In reply to Peter ZijlstraPrior to commit 1cf12e08bc4d ("sched/hotplug: Consolidate task migration on CPU unplug") we'd leave any task on the dying CPU and break affinity and force them off at the very end.
This scheme had to change in order to enable migrate_disable(). One cannot wait for migrate_disable() to complete while stuck in stop_machine(). Furthermore, since we need at the very least: idle, hotplug and stop threads at any point before stop_machine, we can't break affinity and/or push those away.
Under the assumption that all per-cpu kthreads are sanely handled by CPU hotplug, the new code no long breaks affinity or migrates any of them (which then includes the critical ones above).
However, there's an important difference between per-cpu kthreads and kthreads that happen to have a single CPU affinity which is lost. The latter class very much relies on the forced affinity breaking and migration semantics previously provided.
Use the new kthread_is_per_cpu() infrastructure to tighten is_per_cpu_kthread() and fix the hot-unplug problems stemming from the change.
--- kernel/sched/core.c | 29 +++++++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-)
kernel/sched/core.c +25 -4
@@ -1796,13 +1796,28 @@ static inline bool rq_has_pinned_tasks(s*/ static inline bool is_cpu_allowed (struct task_struct *p,int cpu) { +/* When not in the task's cpumask, no point in looking further. */ if (!cpumask_test_cpu (cpu, p->cpus_ptr))return false; -if (is_per_cpu_kthread (p) ||is_migration_disabled (p)) +/* migrate_disabled() must be allowed to finish. */ +if (is_migration_disabled (p))return cpu_online (cpu); -return cpu_active (cpu); +/* Non kernel threads are not allowed during either online or offline. */ +if (!(p->flags &PF_KTHREAD )) +return cpu_active (cpu); + +/* KTHREAD_IS_PER_CPU is always allowed. */ +if (kthread_is_per_cpu (p)) +return cpu_online (cpu); + +/* Regular kernel threads don't get to stay during offline. */ +if (cpu_rq (cpu)->balance_callback == &balance_push_callback) +return cpu_active (cpu); + +/* But are allowed during online. */ +return cpu_online (cpu); }/*
@@ -7276,8 +7291,14 @@ static void balance_push(struct rq *rq)/* * Both the cpu-hotplug and stop task are in this case and are * required to complete the hotplug process. +* +* XXX: the idle task does not match kthread_is_per_cpu() due to +* histerical raisins. */ -if (is_per_cpu_kthread (push_task) ||is_migration_disabled (push_task)) { +if (rq->idle == push_task || + ((push_task->flags &PF_KTHREAD ) &&kthread_is_per_cpu (push_task)) || +is_migration_disabled (push_task)) { +/* * If this is the idle task on the outgoing CPU try to wake * up the hotplug control thread which might wait for the
@@ -7309,7 +7330,7 @@ static void balance_push(struct rq *rq)/* * At this point need_resched() is true and we'll take the loop in * schedule(). The next pick is obviously going to be the stop task -* which is_per_cpu_kthread() and will push this task away. +* which kthread_is_per_cpu() and will push this task away. */ raw_spin_lock (&rq->lock); }
Re: [PATCH 7/8] sched: Fix CPU hotplug / tighten is_per_cpu_kthread()
Quoted 15 lines · Peter Zijlstra
kernel/sched/core.c +4
-2@@ -1796,13 +1796,28 @@ static inline bool rq_has_pinned_tasks(s*/ static inline bool is_cpu_allowed (struct task_struct *p,int cpu) { +/* When not in the task's cpumask, no point in looking further. */ if (!cpumask_test_cpu (cpu, p->cpus_ptr))return false; -if (is_per_cpu_kthread (p) ||is_migration_disabled (p)) +/* migrate_disabled() must be allowed to finish. */ +if (is_migration_disabled (p))return cpu_online (cpu); -return cpu_active (cpu); +/* Non kernel threads are not allowed during either online or offline. */
kernel/sched/core.c static inline bool rq_has_pinned_tasks(s · line 1817
Quoted 10 lines
kernel/sched/core.c +10
-0+if (!(p->flags &PF_KTHREAD )) +return cpu_active (cpu); + +/* KTHREAD_IS_PER_CPU is always allowed. */ +if (kthread_is_per_cpu (p)) +return cpu_online (cpu); + +/* Regular kernel threads don't get to stay during offline. */ +if (cpu_rq (cpu)->balance_callback == &balance_push_callback) +return cpu_active (cpu);
is_cpu_allowed(, cpu) isn't guaranteed to have cpu_rq(cpu)'s rq_lock held, so this can race with balance_push_set(, true). This shouldn't matter under normal circumstances as we'll have sched_cpu_wait_empty() further down the line.
This might get ugly with the rollback faff - this is jumping the gun a bit, but that's something we'll have to address, and I think what I'm concerned about is close to what you mentioned in
http://lore.kernel.org/r/[email protected]
Here's what I'm thinking of:
_cpu_up() ttwu()
select_task_rq()
is_cpu_allowed()
rq->balance_callback != balance_push_callback smpboot_unpark_threads() // FAIL
(now going down, set push here)
sched_cpu_wait_empty()
... ttwu_queue()
sched_cpu_dying()
*ARGH*
I've written some horrors on top of this series here:
https://gitlab.arm.com/linux-arm/linux-vs/-/commits/mainline/migrate_disable/stragglers/
Also, my TX2 is again in need of CPR, so in the meantime I'm running tests on a (much) smaller machine...
Quoted 4 lines
kernel/sched/core.c +3
-0+ +/* But are allowed during online. */ +return cpu_online (cpu); }
Peter Zijlstra
maintainerRe: [PATCH 7/8] sched: Fix CPU hotplug / tighten is_per_cpu_kthread()
Quoted 42 lines · Valentin Schneider
Quoted 13 lines · Peter Zijlstra
(quoted diff) +4
-0@@ -1796,13 +1796,28 @@ static inline bool rq_has_pinned_tasks(s */ static inline bool is_cpu_allowed(struct task_struct *p, int cpu) { + /* When not in the task's cpumask, no point in looking further. */ if (!cpumask_test_cpu(cpu, p->cpus_ptr)) return false; + /* migrate_disabled() must be allowed to finish. */ + if (is_migration_disabled(p)) return cpu_online(cpu); + /* Non kernel threads are not allowed during either online or offline. */Review static inline bool rq_has_pinned_tasks(s · line 1817
Quoted 10 lines
(quoted diff) +10
-0+ if (!(p->flags & PF_KTHREAD)) + return cpu_active(cpu); + + /* KTHREAD_IS_PER_CPU is always allowed. */ + if (kthread_is_per_cpu(p)) + return cpu_online(cpu); + + /* Regular kernel threads don't get to stay during offline. */ + if (cpu_rq(cpu)->balance_callback == &balance_push_callback) + return cpu_active(cpu);
is_cpu_allowed(, cpu)isn't guaranteed to have cpu_rq(cpu)'srq_lockheld, so this can race withbalance_push_set(, true). This shouldn't matter under normal circumstances as we'll havesched_cpu_wait_empty()further down the line.This might get ugly with the rollback faff - this is jumping the gun a bit, but that's something we'll have to address, and I think what I'm concerned about is close to what you mentioned in
http://lore.kernel.org/r/[email protected]
Here's what I'm thinking of:
_cpu_up()ttwu()select_task_rq() is_cpu_allowed() rq->balance_callback != balance_push_callback
Quoted 7 lines
smpboot_unpark_threads()// FAIL
(now going down, set push here)
sched_cpu_wait_empty()
...ttwu_queue()
sched_cpu_dying()
*ARGH*
Let me try this then...
kernel/sched/core.c +13 -0
@@ -7495,6 +7495,8 @@ int sched_cpu_activate(unsigned int cpu)return 0 ; } +unsigned long sched_cpu_rcu_state; +int sched_cpu_deactivate (unsigned int cpu) {struct rq *rq =cpu_rq (cpu);
@@ -7519,6 +7521,11 @@ int sched_cpu_deactivate(unsigned int cpu)*/ balance_push_set (cpu, true); +/* +* See sched_cpu_wait_empty(). +*/ + sched_cpu_rcu_state =get_state_synchronize_rcu (); +rq_lock_irqsave (rq, &rf);if (rq->rd) {update_rq_clock (rq);
@@ -7578,6 +7585,12 @@ int sched_cpu_starting(unsigned int cpu)*/ int sched_cpu_wait_empty (unsigned int cpu) { +/* +* Guarantee that TTWU will observe balance_push_set(true), +* such that all wakeups will refuse this CPU. +*/ +cond_synchronize_rcu (sched_cpu_rcu_state); +balance_hotplug_wait ();return 0 ; }
Peter Zijlstra
maintainerPATCH 8/8 sched: Relax the set_cpus_allowed_ptr() semantics
In reply to Peter ZijlstraNow that we have KTHREAD_IS_PER_CPU to denote the critical per-cpu tasks to retain during CPU offline, we can relax the warning in set_cpus_allowed_ptr(). Any spurious kthread that wants to get on at the last minute will get pushed off before it can run.
While during CPU online there is no harm, and actual benefit, to allowing kthreads back on early, it simplifies hotplug code.
--- kernel/sched/core.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-)
kernel/sched/core.c +9 -11
@@ -2342,7 +2342,9 @@ static int __set_cpus_allowed_ptr(structif (p->flags &PF_KTHREAD ||is_migration_disabled (p)) {/* -* Kernel threads are allowed on online && !active CPUs. +* Kernel threads are allowed on online && !active CPUs, +* however, during cpu-hot-unplug, even these might get pushed +* away if not KTHREAD_IS_PER_CPU. * * Specifically, migration_disabled() tasks must not fail the * cpumask_any_and_distribute() pick below, esp. so on
@@ -2386,16 +2388,6 @@ static int __set_cpus_allowed_ptr(struct__do_set_cpus_allowed (p, new_mask, flags); -if (p->flags &PF_KTHREAD ) { -/* -* For kernel threads that do indeed end up on online && -* !active we want to ensure they are strict per-CPU threads. -*/ -WARN_ON (cpumask_intersects (new_mask, cpu_online_mask) && - !cpumask_intersects (new_mask, cpu_active_mask) && - p->nr_cpus_allowed !=1 ); - } -return affine_move_task (rq, p, &rf, dest_cpu, flags); out:
@@ -7519,6 +7511,12 @@ int sched_cpu_deactivate(unsigned int cp*/ synchronize_rcu (); +/* +* From this point forward, this CPU will refuse to run any task that +* is not: migrate_disable() or KTHREAD_IS_PER_CPU, and will actively +* push those tasks away until this gets cleared, see +* sched_cpu_dying(). +*/ balance_push_set (cpu, true);rq_lock_irqsave (rq, &rf);
Re: [PATCH 8/8] sched: Relax the set_cpus_allowed_ptr() semantics
Quoted 10 lines · Peter Zijlstra
Now that we have
KTHREAD_IS_PER_CPUto denote the critical per-cpu tasks to retain during CPU offline, we can relax the warning inset_cpus_allowed_ptr(). Any spurious kthread that wants to get on at the last minute will get pushed off before it can run.While during CPU online there is no harm, and actual benefit, to allowing kthreads back on early, it simplifies hotplug code.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Thanks!
Relaxing set_cpus_allowed_ptr() was also one of the choices I listed, which can really simplify hotplug code in the workqueue and may be other hotplug code.
Quoted 49 lines
--- kernel/sched/core.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-)kernel/sched/core.c +9
-11@@ -2342,7 +2342,9 @@ static int __set_cpus_allowed_ptr(structif (p->flags &PF_KTHREAD ||is_migration_disabled (p)) {/* -* Kernel threads are allowed on online && !active CPUs. +* Kernel threads are allowed on online && !active CPUs, +* however, during cpu-hot-unplug, even these might get pushed +* away if not KTHREAD_IS_PER_CPU. * * Specifically, migration_disabled() tasks must not fail the * cpumask_any_and_distribute() pick below, esp. so on @@ -2386,16 +2388,6 @@ static int __set_cpus_allowed_ptr(struct__do_set_cpus_allowed (p, new_mask, flags); -if (p->flags &PF_KTHREAD ) { -/* -* For kernel threads that do indeed end up on online && -* !active we want to ensure they are strict per-CPU threads. -*/ -WARN_ON (cpumask_intersects (new_mask, cpu_online_mask) && - !cpumask_intersects (new_mask, cpu_active_mask) && - p->nr_cpus_allowed !=1 ); - } -return affine_move_task (rq, p, &rf, dest_cpu, flags); out:@@ -7519,6 +7511,12 @@ int sched_cpu_deactivate(unsigned int cp*/ synchronize_rcu (); +/* +* From this point forward, this CPU will refuse to run any task that +* is not: migrate_disable() or KTHREAD_IS_PER_CPU, and will actively +* push those tasks away until this gets cleared, see +* sched_cpu_dying(). +*/ balance_push_set (cpu, true);rq_lock_irqsave (rq, &rf);
Peter Zijlstra
maintainerRe: [PATCH 8/8] sched: Relax the set_cpus_allowed_ptr() semantics
Quoted 10 lines · Lai Jiangshan
Quoted 9 lines · Peter Zijlstra
Now that we have
KTHREAD_IS_PER_CPUto denote the critical per-cpu tasks to retain during CPU offline, we can relax the warning inset_cpus_allowed_ptr(). Any spurious kthread that wants to get on at the last minute will get pushed off before it can run.While during CPU online there is no harm, and actual benefit, to allowing kthreads back on early, it simplifies hotplug code.
Quoted 7 lines
Quoted 1 line
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Thanks!
Relaxing
set_cpus_allowed_ptr()was also one of the choices I listed, which can really simplify hotplug code in the workqueue and may be other hotplug code.
Indeed you did. Having that KTHREAD_IS_PER_CPU for the offline side of things made it possible to relax (as per the Changelog above).
Quoted 1 line
Reviewed-by Lai jiangshan <[email protected]>
Thanks!
Peter Zijlstra
maintainerQuoted 7 lines · Peter Zijlstra
Hi,
These patches (no longer 4), seems to fix all the hotplug regressions as per nearly a 100 18*SRCU-P runs over-night.
I did clean up the patches, so possibly I wrecked it again. I've started new runs and will again leave them running over-night.
Hurph... I've got one splat from this version, one I've not seen before:
[ 68.712848] Dying CPU not properly vacated! ... [ 68.744448] CPU1 enqueued tasks (2 total): [ 68.745018] pid: 14, name: rcu_preempt [ 68.745557] pid: 18, name: migration/1
Paul, rcu_preempt, is from rcu_spawn_gp_kthread(), right? Afaict that doesn't even have affinity.. /me wonders HTH that ended up on the runqueue so late.
Quoted 17 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Hi,
These patches (no longer 4), seems to fix all the hotplug regressions as per nearly a 100 18*SRCU-P runs over-night.
I did clean up the patches, so possibly I wrecked it again. I've started new runs and will again leave them running over-night.
Hurph... I've got one splat from this version, one I've not seen before:
[ 68.712848] Dying CPU not properly vacated! ... [ 68.744448] CPU1 enqueued tasks (2 total): [ 68.745018] pid: 14, name: rcu_preempt [ 68.745557] pid: 18, name: migration/1
Quoted 3 lines
Paul,
rcu_preempt, is fromrcu_spawn_gp_kthread(), right? Afaict that doesn't even have affinity.. /me wonders HTH that ended up on the runqueue so late.
Yes, rcu_preempt is from rcu_spawn_gp_kthread(), and you are right that the kernel code does not bind it anywhere. If this is rcutorture, there isn't enough of a userspace to do the binding there, eihter. Wakeups for the rcu_preempt task can happen in odd places, though.
Grasping at straws... Would Frederic's series help? This is in -rcu here:
cfd941c rcu/nocb: Detect unsafe checks for offloaded rdp
028d407 rcu: Remove superfluous rdp fetch
38e216a rcu: Pull deferred rcuog wake up to rcu_eqs_enter() callers 53775fd rcu/nocb: Perform deferred wake up before last idle's need_resched() check 1fbabce rcu/nocb: Trigger self-IPI on late deferred wake up before user resume 2856844 entry: Explicitly flush pending rcuog wakeup before last rescheduling points 4d959df sched: Report local wake up on resched blind zone within idle loop 2617331 entry: Report local wake up on resched blind zone while resuming to user 79acd12 timer: Report ignored local enqueue in nohz mode
I have been including these in all of my tests of your patches.
Thanx, Paul
Peter Zijlstra
maintainerQuoted 22 lines · Paul E. McKenney
Quoted 17 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Hi,
These patches (no longer 4), seems to fix all the hotplug regressions as per nearly a 100 18*SRCU-P runs over-night.
I did clean up the patches, so possibly I wrecked it again. I've started new runs and will again leave them running over-night.
Hurph... I've got one splat from this version, one I've not seen before:
[ 68.712848] Dying CPU not properly vacated! ... [ 68.744448] CPU1 enqueued tasks (2 total): [ 68.745018] pid: 14, name: rcu_preempt [ 68.745557] pid: 18, name: migration/1Quoted 3 lines
Paul,
rcu_preempt, is fromrcu_spawn_gp_kthread(), right? Afaict that doesn't even have affinity.. /me wonders HTH that ended up on the runqueue so late.
Quoted 6 lines
Yes,
rcu_preemptis fromrcu_spawn_gp_kthread(), and you are right that the kernel code does not bind it anywhere. If this is rcutorture, there isn't enough of a userspace to do the binding there, eihter. Wakeups for thercu_preempttask can happen in odd places, though.Grasping at straws...
My current straw is that the wakeup lands on the wakelist before ttwu() will refuse to wake to the CPU, and then lands on the RQ after we've waited. Which seems near impossible..
I'll keep staring..
Quoted 4 lines · Peter Zijlstra
Hi,
These patches (no longer 4), seems to fix all the hotplug regressions as per nearly a 100 18*SRCU-P runs over-night.
Nice!!!
Quoted 4 lines
I did clean up the patches, so possibly I wrecked it again. I've started new runs and will again leave them running over-night.
Paul, if you could please also throw your monster machine at it.
Will do as soon as the tests I started yesterday complete, which should be this afternoon, Pacific Time.
My thought is to do the full set of scenarios overnight, then try hammering either SRCU-P and/or whatever else shows up in the overnight test. Seem reasonable?
Thanx, Paul
Quoted 11 lines · Paul E. McKenney
Quoted 4 lines · Peter Zijlstra
Hi,
These patches (no longer 4), seems to fix all the hotplug regressions as per nearly a 100 18*SRCU-P runs over-night.
Nice!!!
Quoted 3 lines
I did clean up the patches, so possibly I wrecked it again. I've started new runs and will again leave them running over-night.
Quoted 8 lines
Quoted 1 line
Paul, if you could please also throw your monster machine at it.
Will do as soon as the tests I started yesterday complete, which should be this afternoon, Pacific Time.
My thought is to do the full set of scenarios overnight, then try hammering either SRCU-P and/or whatever else shows up in the overnight test. Seem reasonable?
And the SRCU-P runs did fine. I got some task hangs on TREE03 and (to a lesser extent) TREE04. These might well be my fault, so I will try bisecting tomorrow, Pacific Time.
Thanx, Paul