Skip to contentSkip to conversation

sched: Fix hot-unplug regressions

PATCH In review Peter Zijlstra 4 participants 23 messages Jan 12, 2021 – Jan 17, 2021

Subsystems (3)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover sched: Fix hot-unplug regressions

Hi,

These 4 patches are the simplest means (barring a revert) of fixing the CPU hot-unplug problems introduced by commit:

1cf12e08bc4d ("sched/hotplug: Consolidate task migration on CPU unplug")

Testing here, any by Paul, indicate they survive a pounding.

They restore the previous behaviour of forced affinity breaking for the class of kernel threads that happen to have single CPU affinity, but are not strictly a per-cpu kthread.

author maintainer
#1 lore raw

PATCH 2/4 kthread: Extract KTHREAD_IS_PER_CPU

There is a need to distinguish geniune per-cpu kthreads from kthreads that happen to have a single CPU affinity.

Geniune per-cpu kthreads are kthreads that are CPU affine for correctness, these will obviously have PF_KTHREAD set, but must also have PF_NO_SETAFFINITY set, lest userspace modify their affinity and ruins things.

However, these two things are not sufficient, PF_NO_SETAFFINITY is also set on other tasks that have their affinities controlled through other means, like for instance workqueues.

Therefore another bit is needed; it turns out kthread_create_per_cpu() already has such a bit: KTHREAD_IS_PER_CPU, which is used to make kthread_park()/kthread_unpark() work correctly.

Expose this flag and remove the implicit setting of it from kthread_create_on_cpu(); the io_uring usage of it seems dubious at best.

---
 include/linux/kthread.h |    3 +++
 kernel/kthread.c        |   25 ++++++++++++++++++++++++-
 kernel/sched/core.c     |    2 +-
 kernel/sched/sched.h    |    4 ++--
 kernel/smpboot.c        |    1 +
 kernel/workqueue.c      |   11 +++++++++--
 6 files changed, 40 insertions(+), 6 deletions(-)
include/linux/kthread.h +3 -0
@@ -33,6 +33,9 @@ struct task_struct *kthread_create_on_cp
 					  unsigned int cpu,
 					  const char *namefmt);
 
+void kthread_set_per_cpu(struct task_struct *k, bool set);
+bool kthread_is_per_cpu(struct task_struct *k);
+
 /**
  * kthread_run - create and wake a thread.
  * @threadfn: the function to run until signal_pending(current).
kernel/kthread.c +24 -1
@@ -493,11 +493,34 @@ struct task_struct *kthread_create_on_cp
 		return p;
 	kthread_bind(p, cpu);
 	/* CPU hotplug need to bind once again when unparking the thread. */
-	set_bit(KTHREAD_IS_PER_CPU, &to_kthread(p)->flags);
 	to_kthread(p)->cpu = cpu;
 	return p;
 }
 
+void kthread_set_per_cpu(struct task_struct *k, bool set)
+{
+	struct kthread *kthread = to_kthread(k);
+	if (!kthread)
+		return;
+
+	if (set) {
+		WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
+		WARN_ON_ONCE(k->nr_cpus_allowed != 1);
+		set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
+	} else {
+		clear_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
+	}
+}
+
+bool kthread_is_per_cpu(struct task_struct *k)
+{
+	struct kthread *kthread = to_kthread(k);
+	if (!kthread)
+		return false;
+
+	return test_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
+}
+
 /**
  * kthread_unpark - unpark a thread created by kthread_create().
  * @k:		thread created by kthread_create().
kernel/smpboot.c +1 -0
@@ -188,6 +188,7 @@ __smpboot_create_thread(struct smp_hotpl
 		kfree(td);
 		return PTR_ERR(tsk);
 	}
+	kthread_set_per_cpu(tsk, true);
 	/*
 	 * Park the thread so that it could start right on the CPU
 	 * when it is available.
author maintainer
#2 lore raw

PATCH 3/4 workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

In reply to Peter Zijlstra

Mark the per-cpu workqueue workers as KTHREAD_IS_PER_CPU.

Workqueues have unfortunate semantics in that per-cpu workers are not default flushed and parked during hotplug, however a subset does manual flush on hotplug and hard relies on them for correctness.

Therefore play silly games..

---
 kernel/workqueue.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/workqueue.c +9 -2
@@ -1861,6 +1861,8 @@ static void worker_attach_to_pool(struct
 	 */
 	if (pool->flags & POOL_DISASSOCIATED)
 		worker->flags |= WORKER_UNBOUND;
+	else
+		kthread_set_per_cpu(worker->task, true);
 
 	list_add_tail(&worker->node, &pool->workers);
 	worker->pool = pool;
@@ -1883,6 +1885,7 @@ static void worker_detach_from_pool(stru
 
 	mutex_lock(&wq_pool_attach_mutex);
 
+	kthread_set_per_cpu(worker->task, false);
 	list_del(&worker->node);
 	worker->pool = NULL;
 
@@ -4919,8 +4922,10 @@ static void unbind_workers(int cpu)
 
 		raw_spin_unlock_irq(&pool->lock);
 
-		for_each_pool_worker(worker, pool)
+		for_each_pool_worker(worker, pool) {
+			kthread_set_per_cpu(worker->task, false);
 			WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
+		}
 
 		mutex_unlock(&wq_pool_attach_mutex);
 
@@ -4972,9 +4977,11 @@ static void rebind_workers(struct worker
 	 * of all workers first and then clear UNBOUND.  As we're called
 	 * from CPU_ONLINE, the following shouldn't fail.
 	 */
-	for_each_pool_worker(worker, pool)
+	for_each_pool_worker(worker, pool) {
 		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
 						  pool->attrs->cpumask) < 0);
+		kthread_set_per_cpu(worker->task, true);
+	}
 
 	raw_spin_lock_irq(&pool->lock);
 
reviewer Reviewed-by on v1 3/4
#3 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 7 lines · Peter Zijlstra

Mark the per-cpu workqueue workers as KTHREAD_IS_PER_CPU.

Workqueues have unfortunate semantics in that per-cpu workers are not default flushed and parked during hotplug, however a subset does manual flush on hotplug and hard relies on them for correctness.

Quoted 5 lines

Therefore play silly games..

---

I like this patchset in that the scheduler takes care of the affinities of the tasks when we don't want it to be per-cpu.

Quoted 49 lines
 kernel/workqueue.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/workqueue.c +9 -2
@@ -1861,6 +1861,8 @@ static void worker_attach_to_pool(struct
         */
        if (pool->flags & POOL_DISASSOCIATED)
                worker->flags |= WORKER_UNBOUND;
+       else
+               kthread_set_per_cpu(worker->task, true);
 
        list_add_tail(&worker->node, &pool->workers);
        worker->pool = pool;
@@ -1883,6 +1885,7 @@ static void worker_detach_from_pool(stru
 
        mutex_lock(&wq_pool_attach_mutex);
 
+       kthread_set_per_cpu(worker->task, false);
        list_del(&worker->node);
        worker->pool = NULL;
 
@@ -4919,8 +4922,10 @@ static void unbind_workers(int cpu)
 
                raw_spin_unlock_irq(&pool->lock);
 
-               for_each_pool_worker(worker, pool)
+               for_each_pool_worker(worker, pool) {
+                       kthread_set_per_cpu(worker->task, false);
                        WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
+               }
 
                mutex_unlock(&wq_pool_attach_mutex);
 
@@ -4972,9 +4977,11 @@ static void rebind_workers(struct worker
         * of all workers first and then clear UNBOUND.  As we're called
         * from CPU_ONLINE, the following shouldn't fail.
         */
-       for_each_pool_worker(worker, pool)
+       for_each_pool_worker(worker, pool) {
                WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
                                                  pool->attrs->cpumask) < 0);
+               kthread_set_per_cpu(worker->task, true);
+       }
 
        raw_spin_lock_irq(&pool->lock);
 
author maintainer
#4 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 10 lines · Lai Jiangshan
Quoted 9 lines · Peter Zijlstra

Mark the per-cpu workqueue workers as KTHREAD_IS_PER_CPU.

Workqueues have unfortunate semantics in that per-cpu workers are not default flushed and parked during hotplug, however a subset does manual flush on hotplug and hard relies on them for correctness.

Therefore play silly games..

Quoted 8 lines
Quoted 3 lines

---

I like this patchset in that the scheduler takes care of the affinities of the tasks when we don't want it to be per-cpu.

Thanks! A possibly even simpler approach would be to have rebind_workers() kill all workers and have create_worker() spawn us new ones.

That avoids ever having to use set_cpus_allowed_ptr() on per-cpu kthreads.... with the exception of rescuer.. still pondering that.

Reviewed-by on v1 3/4
#5 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

In reply to Peter Zijlstra

kernel/workqueue.c static void unbind_workers(int cpu) · line 4928

Quoted 9 lines · Peter Zijlstra
kernel/workqueue.c +3 -1
@@ -4919,8 +4922,10 @@ static void unbind_workers(int cpu)
 
              raw_spin_unlock_irq(&pool->lock);
 
-		for_each_pool_worker(worker, pool)
+		for_each_pool_worker(worker, pool) {
+			kthread_set_per_cpu(worker->task, false);
                      WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
+		}

Doesn't this supersede patch 1? With patch 4 on top, the BALANCE_PUSH stuff should start resetting the affinity of the kworkers for which we are removing the IS_PER_CPU flag.

It's the only nit I have, the rest looks good to me so:

I'll go frob that sched_cpu_dying() warning.

reviewer
#6 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

In reply to Peter Zijlstra
Quoted 47 lines · Peter Zijlstra

Mark the per-cpu workqueue workers as KTHREAD_IS_PER_CPU.

Workqueues have unfortunate semantics in that per-cpu workers are not default flushed and parked during hotplug, however a subset does manual flush on hotplug and hard relies on them for correctness.

Therefore play silly games..

---
 kernel/workqueue.c |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/workqueue.c +6 -1
@@ -1861,6 +1861,8 @@ static void worker_attach_to_pool(struct
         */
        if (pool->flags & POOL_DISASSOCIATED)
                worker->flags |= WORKER_UNBOUND;
+       else
+               kthread_set_per_cpu(worker->task, true);
 
        list_add_tail(&worker->node, &pool->workers);
        worker->pool = pool;
@@ -1883,6 +1885,7 @@ static void worker_detach_from_pool(stru
 
        mutex_lock(&wq_pool_attach_mutex);
 
+       kthread_set_per_cpu(worker->task, false);
        list_del(&worker->node);
        worker->pool = NULL;
 
@@ -4919,8 +4922,10 @@ static void unbind_workers(int cpu)
 
                raw_spin_unlock_irq(&pool->lock);
 
-               for_each_pool_worker(worker, pool)
+               for_each_pool_worker(worker, pool) {
+                       kthread_set_per_cpu(worker->task, false);
                        WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
+               }
 
                mutex_unlock(&wq_pool_attach_mutex);
 

kernel/workqueue.c static void rebind_workers(struct worker · line 4983

Quoted 8 lines
kernel/workqueue.c +2 -1
@@ -4972,9 +4977,11 @@ static void rebind_workers(struct worker
         * of all workers first and then clear UNBOUND.  As we're called
         * from CPU_ONLINE, the following shouldn't fail.
         */
-       for_each_pool_worker(worker, pool)
+       for_each_pool_worker(worker, pool) {
                WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
                                                  pool->attrs->cpumask) < 0);
+               kthread_set_per_cpu(worker->task, true);

Will the schedule break affinity in the middle of these two lines due to patch4 allowing it and result in Paul's reported splat.

Quoted 6 lines
kernel/workqueue.c +1 -0
+       }
 
        raw_spin_lock_irq(&pool->lock);
 
#7 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 14 lines · Lai Jiangshan

kernel/workqueue.c static void rebind_workers(struct worker · line 4983

Quoted 9 lines · Peter Zijlstra
kernel/workqueue.c +2 -1
@@ -4972,9 +4977,11 @@ static void rebind_workers(struct worker
         * of all workers first and then clear UNBOUND.  As we're called
         * from CPU_ONLINE, the following shouldn't fail.
         */
-       for_each_pool_worker(worker, pool)
+       for_each_pool_worker(worker, pool) {
                WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
                                                  pool->attrs->cpumask) < 0);
+               kthread_set_per_cpu(worker->task, true);

Will the schedule break affinity in the middle of these two lines due to patch4 allowing it and result in Paul's reported splat.

You might be right; at this point we would still have BALANCE_PUSH set, so something like the below could happen

rebind_workers()

    set_cpus_allowed_ptr()
      affine_move_task()
        task_running() => stop_one_cpu()

... // Stopper migrates the kworker here in the meantime

switch_to(<pcpu kworker>) // Both cpuhp thread and kworker should be enqueued

                            // here, so one or the other could be picked
  balance_switch()
    balance_push()
    ^-- no KTHREAD_IS_PER_CPU !

This should however trigger the WARN_ON_ONCE() in kthread_set_per_cpu() *before* the one in process_one_work(), which I haven't seen in Paul's mails.

Quoted 6 lines
Quoted 6 lines
kernel/workqueue.c +1 -0
+       }
 
        raw_spin_lock_irq(&pool->lock);
 
#8 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 27 lines · Valentin Schneider
Quoted 14 lines · Lai Jiangshan

kernel/workqueue.c static void rebind_workers(struct worker · line 4983

Quoted 9 lines · Peter Zijlstra
kernel/workqueue.c +2 -1
@@ -4972,9 +4977,11 @@ static void rebind_workers(struct worker
         * of all workers first and then clear UNBOUND.  As we're called
         * from CPU_ONLINE, the following shouldn't fail.
         */
-       for_each_pool_worker(worker, pool)
+       for_each_pool_worker(worker, pool) {
                WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
                                                  pool->attrs->cpumask) < 0);
+               kthread_set_per_cpu(worker->task, true);

Will the schedule break affinity in the middle of these two lines due to patch4 allowing it and result in Paul's reported splat.

You might be right; at this point we would still have BALANCE_PUSH set, so something like the below could happen

rebind_workers()

    set_cpus_allowed_ptr()
      affine_move_task()
        task_running() => stop_one_cpu()

... // Stopper migrates the kworker here in the meantime

switch_to(<pcpu kworker>) // Both cpuhp thread and kworker should be enqueued

Quoted 8 lines
                            // here, so one or the other could be picked
  balance_switch()
    balance_push()
    ^-- no KTHREAD_IS_PER_CPU !

This should however trigger the WARN_ON_ONCE() in kthread_set_per_cpu() *before* the one in process_one_work(), which I haven't seen in Paul's mails.

The 56 instances of one-hour SRCU-P scenarios hit the WARN_ON_ONCE() in process_one_work() once, but there is no sign of a WARN_ON_ONCE() from kthread_set_per_cpu(). But to your point, this does appear to be a rather low-probability race condition, once per some tens of hours of SRCU-P.

Is there a more focused check for the race condition above?

Thanx, Paul

#9 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 17 lines · Paul E. McKenney
Quoted 16 lines · Valentin Schneider

You might be right; at this point we would still have BALANCE_PUSH set, so something like the below could happen

rebind_workers()

    set_cpus_allowed_ptr()
      affine_move_task()
        task_running() => stop_one_cpu()

... // Stopper migrates the kworker here in the meantime

switch_to(<pcpu kworker>) // Both cpuhp thread and kworker should be enqueued

                            // here, so one or the other could be picked
  balance_switch()
    balance_push()
    ^-- no KTHREAD_IS_PER_CPU !
Quoted 7 lines
Quoted 3 lines

This should however trigger the WARN_ON_ONCE() in kthread_set_per_cpu() *before* the one in process_one_work(), which I haven't seen in Paul's mails.

The 56 instances of one-hour SRCU-P scenarios hit the WARN_ON_ONCE() in process_one_work() once, but there is no sign of a WARN_ON_ONCE() from kthread_set_per_cpu().

This does make me doubt the above :/ At the same time, the process_one_work() warning hinges on POOL_DISASSOCIATED being unset, which implies having gone through rebind_workers(), which implies kthread_set_per_cpu(), which implies me being quite confused...

Quoted 6 lines

But to your point, this does appear to be
a rather low-probability race condition, once per some tens of hours of SRCU-P.

Is there a more focused check for the race condition above?

Not that I'm aware of. I'm thinking that if the pcpu kworker were an RT task, then this would guarantee it would get picked in favor of the cpuhp thread upon switching out of the stopper, but that still requires the kworker running on some CPU (for some reason) during rebind_workers().

Quoted 1 line

Thanx, Paul

#10 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 35 lines · Valentin Schneider
Quoted 17 lines · Paul E. McKenney
Quoted 16 lines · Valentin Schneider

You might be right; at this point we would still have BALANCE_PUSH set, so something like the below could happen

rebind_workers()

    set_cpus_allowed_ptr()
      affine_move_task()
        task_running() => stop_one_cpu()

... // Stopper migrates the kworker here in the meantime

switch_to(<pcpu kworker>) // Both cpuhp thread and kworker should be enqueued

                            // here, so one or the other could be picked
  balance_switch()
    balance_push()
    ^-- no KTHREAD_IS_PER_CPU !
Quoted 7 lines
Quoted 3 lines

This should however trigger the WARN_ON_ONCE() in kthread_set_per_cpu() *before* the one in process_one_work(), which I haven't seen in Paul's mails.

The 56 instances of one-hour SRCU-P scenarios hit the WARN_ON_ONCE() in process_one_work() once, but there is no sign of a WARN_ON_ONCE() from kthread_set_per_cpu().

This does make me doubt the above :/ At the same time, the process_one_work() warning hinges on POOL_DISASSOCIATED being unset, which implies having gone through rebind_workers(), which implies kthread_set_per_cpu(), which implies me being quite confused...

Quoted 4 lines

But to your point, this does appear to be
a rather low-probability race condition, once per some tens of hours of SRCU-P.

Quoted 6 lines
Quoted 1 line

Is there a more focused check for the race condition above?

Not that I'm aware of. I'm thinking that if the pcpu kworker were an RT task, then this would guarantee it would get picked in favor of the cpuhp thread upon switching out of the stopper, but that still requires the kworker running on some CPU (for some reason) during rebind_workers().

Well, I did use the rcutree.softirq=0 boot parameter, which creates per-CPU rcuc kthreads to do what RCU_SOFTIRQ normally does. But these rcuc kthreads use the normal park/unpark discipline, so should be safe, for some value of "should".

Thanx, Paul

author maintainer
#11 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

In reply to Lai Jiangshan
Quoted 13 lines · Lai Jiangshan

Review static void rebind_workers(struct worker · line 4983

Quoted 9 lines · Peter Zijlstra
(quoted diff) +2 -1
@@ -4972,9 +4977,11 @@ static void rebind_workers(struct worker
         * of all workers first and then clear UNBOUND.  As we're called
         * from CPU_ONLINE, the following shouldn't fail.
         */
-       for_each_pool_worker(worker, pool)
+       for_each_pool_worker(worker, pool) {
                WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
                                                  pool->attrs->cpumask) < 0);
+               kthread_set_per_cpu(worker->task, true);

Will the schedule break affinity in the middle of these two lines due to patch4 allowing it and result in Paul's reported splat.

So something like the below _should_ work, except i'm seeing odd WARNs. I'll prod at it some more.

kernel/workqueue.c +4 -2
@@ -2371,6 +2371,7 @@ static int worker_thread(void *__worker)
 	/* tell the scheduler that this is a workqueue worker */
 	set_pf_worker(true);
 woke_up:
+	kthread_parkme();
 	raw_spin_lock_irq(&pool->lock);
 
 	/* am I supposed to die? */
@@ -2428,6 +2429,7 @@ static int worker_thread(void *__worker)
 			move_linked_works(work, &worker->scheduled, NULL);
 			process_scheduled_works(worker);
 		}
+		kthread_parkme();
 	} while (keep_working(pool));
 
 	worker_set_flags(worker, WORKER_PREP);
@@ -4978,9 +4980,9 @@ static void rebind_workers(struct worker
 	 * from CPU_ONLINE, the following shouldn't fail.
 	 */
 	for_each_pool_worker(worker, pool) {
-		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-						  pool->attrs->cpumask) < 0);
+		kthread_park(worker->task);
 		kthread_set_per_cpu(worker->task, true);
+		kthread_unpark(worker->task);
 	}
 
 	raw_spin_lock_irq(&pool->lock);
#12 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 37 lines · Peter Zijlstra
Quoted 13 lines · Lai Jiangshan

Review static void rebind_workers(struct worker · line 4983

Quoted 9 lines · Peter Zijlstra
(quoted diff) +2 -1
@@ -4972,9 +4977,11 @@ static void rebind_workers(struct worker
         * of all workers first and then clear UNBOUND.  As we're called
         * from CPU_ONLINE, the following shouldn't fail.
         */
-       for_each_pool_worker(worker, pool)
+       for_each_pool_worker(worker, pool) {
                WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
                                                  pool->attrs->cpumask) < 0);
+               kthread_set_per_cpu(worker->task, true);

Will the schedule break affinity in the middle of these two lines due to patch4 allowing it and result in Paul's reported splat.

So something like the below _should_ work, except i'm seeing odd WARNs. I'll prod at it some more.

kernel/workqueue.c +2 -0
@@ -2371,6 +2371,7 @@ static int worker_thread(void *__worker)
      /* tell the scheduler that this is a workqueue worker */
      set_pf_worker(true);
 woke_up:
+	kthread_parkme();
      raw_spin_lock_irq(&pool->lock);
 
      /* am I supposed to die? */
@@ -2428,6 +2429,7 @@ static int worker_thread(void *__worker)
                      move_linked_works(work, &worker->scheduled, NULL);
                      process_scheduled_works(worker);
              }
+		kthread_parkme();
      } while (keep_working(pool));
 
      worker_set_flags(worker, WORKER_PREP);

kernel/workqueue.c static void rebind_workers(struct worker · line 4983

Quoted 6 lines
kernel/workqueue.c +1 -2
@@ -4978,9 +4980,9 @@ static void rebind_workers(struct worker
       * from CPU_ONLINE, the following shouldn't fail.
       */
      for_each_pool_worker(worker, pool) {
-		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-						  pool->attrs->cpumask) < 0);
+		kthread_park(worker->task);

Don't we still need an affinity change here, to undo what was done in unbind_workers()?

Would something like

__kthread_bind_mask(worker->task, pool->attrs->cpumask, TASK_PARKED)

even work?

Quoted 5 lines
kernel/workqueue.c +1 -0
              kthread_set_per_cpu(worker->task, true);
+		kthread_unpark(worker->task);
      }
 
      raw_spin_lock_irq(&pool->lock);
author maintainer
#13 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 9 lines · Valentin Schneider

Review

Quoted 3 lines · Peter Zijlstra
(quoted diff) +1 -2
-		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-						  pool->attrs->cpumask) < 0);
+		kthread_park(worker->task);

Don't we still need an affinity change here, to undo what was done in unbind_workers()?

Quoted 8 lines

Would something like

__kthread_bind_mask(worker->task, pool->attrs->cpumask, TASK_PARKED)

even work?

Review

Quoted 2 lines
              kthread_set_per_cpu(worker->task, true);
(quoted diff) +1 -0
+		kthread_unpark(worker->task);

Nope, look at what kthread_unpark() does, what was missing was assigning kthread->cpu though.

The below seems to actually work. Rescuer is still a problem though.

---
 include/linux/kthread.h |  2 +-
 kernel/kthread.c        | 14 ++++++++------
 kernel/sched/core.c     | 19 ++++++++++++++++++-
 kernel/smpboot.c        |  2 +-
 kernel/workqueue.c      | 22 +++++++++++++---------
 5 files changed, 41 insertions(+), 18 deletions(-)
include/linux/kthread.h +1 -1
index fdd5a52e35d8..2484ed97e72f 100644
@@ -33,7 +33,7 @@ struct task_struct *kthread_create_on_cpu(int (*threadfn)(void *data),
 					  unsigned int cpu,
 					  const char *namefmt);
 
-void kthread_set_per_cpu(struct task_struct *k, bool set);
+void kthread_set_per_cpu(struct task_struct *k, int cpu);
 bool kthread_is_per_cpu(struct task_struct *k);
 
 /**
kernel/kthread.c +8 -6
index bead90275d2b..e0e4a423f184 100644
@@ -497,19 +497,21 @@ struct task_struct *kthread_create_on_cpu(int (*threadfn)(void *data),
 	return p;
 }
 
-void kthread_set_per_cpu(struct task_struct *k, bool set)
+void kthread_set_per_cpu(struct task_struct *k, int cpu)
 {
 	struct kthread *kthread = to_kthread(k);
 	if (!kthread)
 		return;
 
-	if (set) {
-		WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
-		WARN_ON_ONCE(k->nr_cpus_allowed != 1);
-		set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
-	} else {
+	WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
+
+	if (cpu < 0) {
 		clear_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
+		return;
 	}
+
+	kthread->cpu = cpu;
+	set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
 }
 
 bool kthread_is_per_cpu(struct task_struct *k)
kernel/sched/core.c +18 -1
index 60b257d845fa..c2fdeeb6af2b 100644
@@ -7589,7 +7589,24 @@ int sched_cpu_dying(unsigned int cpu)
 	sched_tick_stop(cpu);
 
 	rq_lock_irqsave(rq, &rf);
-	BUG_ON(rq->nr_running != 1 || rq_has_pinned_tasks(rq));
+	if (rq->nr_running != 1 || rq_has_pinned_tasks(rq)) {
+		struct task_struct *g, *p;
+
+		pr_crit("CPU%d nr_running=%d\n", cpu, rq->nr_running);
+		rcu_read_lock();
+		for_each_process_thread(g, p) {
+			if (task_cpu(p) != cpu)
+				continue;
+
+			if (!task_on_rq_queued(p))
+				continue;
+
+			pr_crit("\tp=%s\n", p->comm);
+		}
+		rcu_read_unlock();
+
+		WARN_ON_ONCE(1);
+	}
 	rq_unlock_irqrestore(rq, &rf);
 
 	calc_load_migrate(rq);
kernel/smpboot.c +1 -1
index b0abe575a524..f25208e8df83 100644
@@ -188,7 +188,7 @@ __smpboot_create_thread(struct smp_hotplug_thread *ht, unsigned int cpu)
 		kfree(td);
 		return PTR_ERR(tsk);
 	}
-	kthread_set_per_cpu(tsk, true);
+	kthread_set_per_cpu(tsk, cpu);
 	/*
 	 * Park the thread so that it could start right on the CPU
 	 * when it is available.
kernel/workqueue.c +13 -9
index ec0771e4a3fb..b518fd67a792 100644
@@ -1862,7 +1862,7 @@ static void worker_attach_to_pool(struct worker *worker,
 	if (pool->flags & POOL_DISASSOCIATED)
 		worker->flags |= WORKER_UNBOUND;
 	else
-		kthread_set_per_cpu(worker->task, true);
+		kthread_set_per_cpu(worker->task, pool->cpu);
 
 	list_add_tail(&worker->node, &pool->workers);
 	worker->pool = pool;
@@ -1885,7 +1885,7 @@ static void worker_detach_from_pool(struct worker *worker)
 
 	mutex_lock(&wq_pool_attach_mutex);
 
-	kthread_set_per_cpu(worker->task, false);
+	kthread_set_per_cpu(worker->task, -1);
 	list_del(&worker->node);
 	worker->pool = NULL;
 
@@ -2371,6 +2371,7 @@ static int worker_thread(void *__worker)
 	/* tell the scheduler that this is a workqueue worker */
 	set_pf_worker(true);
 woke_up:
+	kthread_parkme();
 	raw_spin_lock_irq(&pool->lock);
 
 	/* am I supposed to die? */
@@ -2428,7 +2429,7 @@ static int worker_thread(void *__worker)
 			move_linked_works(work, &worker->scheduled, NULL);
 			process_scheduled_works(worker);
 		}
-	} while (keep_working(pool));
+	} while (keep_working(pool) && !kthread_should_park());
 
 	worker_set_flags(worker, WORKER_PREP);
 sleep:
@@ -2440,9 +2441,12 @@ static int worker_thread(void *__worker)
 	 * event.
 	 */
 	worker_enter_idle(worker);
-	__set_current_state(TASK_IDLE);
+	set_current_state(TASK_IDLE);
 	raw_spin_unlock_irq(&pool->lock);
-	schedule();
+
+	if (!kthread_should_park())
+		schedule();
+
 	goto woke_up;
 }
 
@@ -4923,7 +4927,7 @@ static void unbind_workers(int cpu)
 		raw_spin_unlock_irq(&pool->lock);
 
 		for_each_pool_worker(worker, pool) {
-			kthread_set_per_cpu(worker->task, false);
+			kthread_set_per_cpu(worker->task, -1);
 			WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
 		}
 
@@ -4978,9 +4982,9 @@ static void rebind_workers(struct worker_pool *pool)
 	 * from CPU_ONLINE, the following shouldn't fail.
 	 */
 	for_each_pool_worker(worker, pool) {
-		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-						  pool->attrs->cpumask) < 0);
-		kthread_set_per_cpu(worker->task, true);
+		WARN_ON_ONCE(kthread_park(worker->task) < 0);
+		kthread_set_per_cpu(worker->task, pool->cpu);
+		kthread_unpark(worker->task);
 	}
 
 	raw_spin_lock_irq(&pool->lock);
reviewer
#14 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 14 lines · Peter Zijlstra
kernel/kthread.c +1 -6
-void kthread_set_per_cpu(struct task_struct *k, bool set)
+void kthread_set_per_cpu(struct task_struct *k, int cpu)
 {
        struct kthread *kthread = to_kthread(k);
        if (!kthread)
                return;
 
-       if (set) {
-               WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
-               WARN_ON_ONCE(k->nr_cpus_allowed != 1);
-               set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
-       } else {

kernel/kthread.c struct task_struct *kthread_create_on_cpu(int (*threadfn)(void *data), · line 515

Quoted 10 lines
kernel/kthread.c +7 -0
+       WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
+
+       if (cpu < 0) {
                clear_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
+               return;
        }
+
+       kthread->cpu = cpu;
+       set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
 }

I don't see the code to set the mask of the cpu to the task since set_cpus_allowed_ptr() is removed from rebind_worker().

Is it somewhere I missed?

Quoted 43 lines
kernel/workqueue.c +8 -4
@@ -2371,6 +2371,7 @@ static int worker_thread(void *__worker)
        /* tell the scheduler that this is a workqueue worker */
        set_pf_worker(true);
 woke_up:
+       kthread_parkme();
        raw_spin_lock_irq(&pool->lock);
 
        /* am I supposed to die? */
@@ -2428,7 +2429,7 @@ static int worker_thread(void *__worker)
                        move_linked_works(work, &worker->scheduled, NULL);
                        process_scheduled_works(worker);
                }
-       } while (keep_working(pool));
+       } while (keep_working(pool) && !kthread_should_park());
 
        worker_set_flags(worker, WORKER_PREP);
 sleep:
@@ -2440,9 +2441,12 @@ static int worker_thread(void *__worker)
         * event.
         */
        worker_enter_idle(worker);
-       __set_current_state(TASK_IDLE);
+       set_current_state(TASK_IDLE);
        raw_spin_unlock_irq(&pool->lock);
-       schedule();
+
+       if (!kthread_should_park())
+               schedule();
+
        goto woke_up;
 }
 
@@ -4923,7 +4927,7 @@ static void unbind_workers(int cpu)
                raw_spin_unlock_irq(&pool->lock);
 
                for_each_pool_worker(worker, pool) {
-                       kthread_set_per_cpu(worker->task, false);
+                       kthread_set_per_cpu(worker->task, -1);
                        WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, cpu_possible_mask) < 0);
                }
 

kernel/workqueue.c static void rebind_workers(struct worker_pool *pool) · line 4987

Quoted 9 lines
kernel/workqueue.c +3 -3
@@ -4978,9 +4982,9 @@ static void rebind_workers(struct worker_pool *pool)
         * from CPU_ONLINE, the following shouldn't fail.
         */
        for_each_pool_worker(worker, pool) {
-               WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-                                                 pool->attrs->cpumask) < 0);
-               kthread_set_per_cpu(worker->task, true);
+               WARN_ON_ONCE(kthread_park(worker->task) < 0);
+               kthread_set_per_cpu(worker->task, pool->cpu);
+               kthread_unpark(worker->task);

I feel nervous to use kthread_park() here and kthread_parkme() in worker thread. And adding kthread_should_park() to the fast path also daunt me.

How about using a new KTHREAD_XXXX instead of KTHREAD_IS_PER_CPU, so that we can set and clear KTHREAD_XXXX freely, especially before set_cpus_allowed_ptr().

For example, we can add a new KTHREAD_ACTIVE_MASK_ONLY which means even when
is_per_cpu_kthread() && the_cpu_is_online() && the_cpu_is_not_active() && KTHREAD_ACTIVE_MASK_ONLY
we should also break the affinity.

So that we can easily set KTHREAD_ACTIVE_MASK_ONLY in unbind_workers() add clear KTHREAD_ACTIVE_MASK_ONLY here and avoid adding new synchronization like kthread_park().

Quoted 3 lines
kernel/workqueue.c +0 -0
        }
 
        raw_spin_lock_irq(&pool->lock);
author maintainer
#15 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 30 lines · Lai Jiangshan
Quoted 14 lines · Peter Zijlstra
kernel/kthread.c +1 -6
-void kthread_set_per_cpu(struct task_struct *k, bool set)
+void kthread_set_per_cpu(struct task_struct *k, int cpu)
 {
        struct kthread *kthread = to_kthread(k);
        if (!kthread)
                return;
 
-       if (set) {
-               WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
-               WARN_ON_ONCE(k->nr_cpus_allowed != 1);
-               set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
-       } else {

kernel/kthread.c struct task_struct *kthread_create_on_cpu(int (*threadfn)(void *data), · line 515

Quoted 10 lines
kernel/kthread.c +7 -0
+       WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
+
+       if (cpu < 0) {
                clear_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
+               return;
        }
+
+       kthread->cpu = cpu;
+       set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
 }

I don't see the code to set the mask of the cpu to the task since set_cpus_allowed_ptr() is removed from rebind_worker().

Quoted 1 line

Is it somewhere I missed?

kthread_unpark().

Quoted 6 lines
Quoted 6 lines
kernel/workqueue.c +0 -2
@@ -4978,9 +4982,9 @@ static void rebind_workers(struct worker_pool *pool)
         * from CPU_ONLINE, the following shouldn't fail.
         */
        for_each_pool_worker(worker, pool) {
-               WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-                                                 pool->attrs->cpumask) < 0);
Quoted 8 lines

kernel/workqueue.c static void rebind_workers(struct worker_pool *pool) · line 4987

Quoted 4 lines
kernel/workqueue.c +3 -1
-               kthread_set_per_cpu(worker->task, true);
+               WARN_ON_ONCE(kthread_park(worker->task) < 0);
+               kthread_set_per_cpu(worker->task, pool->cpu);
+               kthread_unpark(worker->task);

I feel nervous to use kthread_park() here and kthread_parkme() in worker thread. And adding kthread_should_park() to the fast path also daunt me.

Is that really such a hot path that an additional load is problematic?

Quoted 3 lines

How about using a new KTHREAD_XXXX instead of KTHREAD_IS_PER_CPU, so that we can set and clear KTHREAD_XXXX freely, especially before set_cpus_allowed_ptr().

KTHREAD_IS_PER_CPU is exactly what we need, why make another flag?

The above sequence is nice in that it restores both the KTHREAD_IS_PER_CPU flag and affinity while the task is frozen, so there are no races where one is observed and not the other.

It is also the exact sequence normal per-cpu threads (smpboot) use to preserve affinity.

reviewer
#16 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 57 lines · Peter Zijlstra
Quoted 30 lines · Lai Jiangshan
Quoted 14 lines · Peter Zijlstra
kernel/kthread.c +1 -6
-void kthread_set_per_cpu(struct task_struct *k, bool set)
+void kthread_set_per_cpu(struct task_struct *k, int cpu)
 {
        struct kthread *kthread = to_kthread(k);
        if (!kthread)
                return;
 
-       if (set) {
-               WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
-               WARN_ON_ONCE(k->nr_cpus_allowed != 1);
-               set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
-       } else {

kernel/kthread.c struct task_struct *kthread_create_on_cpu(int (*threadfn)(void *data), · line 515

Quoted 10 lines
kernel/kthread.c +7 -0
+       WARN_ON_ONCE(!(k->flags & PF_NO_SETAFFINITY));
+
+       if (cpu < 0) {
                clear_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
+               return;
        }
+
+       kthread->cpu = cpu;
+       set_bit(KTHREAD_IS_PER_CPU, &kthread->flags);
 }

I don't see the code to set the mask of the cpu to the task since set_cpus_allowed_ptr() is removed from rebind_worker().

Quoted 1 line

Is it somewhere I missed?

kthread_unpark().

Quoted 6 lines
Quoted 6 lines
kernel/workqueue.c +0 -2
@@ -4978,9 +4982,9 @@ static void rebind_workers(struct worker_pool *pool)
         * from CPU_ONLINE, the following shouldn't fail.
         */
        for_each_pool_worker(worker, pool) {
-               WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-                                                 pool->attrs->cpumask) < 0);
Quoted 8 lines

kernel/workqueue.c static void rebind_workers(struct worker_pool *pool) · line 4987

Quoted 4 lines
kernel/workqueue.c +3 -1
-               kthread_set_per_cpu(worker->task, true);
+               WARN_ON_ONCE(kthread_park(worker->task) < 0);
+               kthread_set_per_cpu(worker->task, pool->cpu);
+               kthread_unpark(worker->task);

I feel nervous to use kthread_park() here and kthread_parkme() in worker thread. And adding kthread_should_park() to the fast path also daunt me.

Is that really such a hot path that an additional load is problematic?

Quoted 3 lines

How about using a new KTHREAD_XXXX instead of KTHREAD_IS_PER_CPU, so that we can set and clear KTHREAD_XXXX freely, especially before set_cpus_allowed_ptr().

Quoted 8 lines

KTHREAD_IS_PER_CPU is exactly what we need, why make another flag?

The above sequence is nice in that it restores both the KTHREAD_IS_PER_CPU flag and affinity while the task is frozen, so there are no races where one is observed and not the other.

It is also the exact sequence normal per-cpu threads (smpboot) use to preserve affinity.

Other per-cpu threads normally do short-live works. wq's work can be lengthy, cpu-intensive, heavy-lock-acquiring or even call get_online_cpus() which might result in a deadlock with kthread_park().

author maintainer
#17 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 7 lines · Lai Jiangshan
Quoted 2 lines · Peter Zijlstra

It is also the exact sequence normal per-cpu threads (smpboot) use to preserve affinity.

Other per-cpu threads normally do short-live works. wq's work can be lengthy, cpu-intensive, heavy-lock-acquiring or even call get_online_cpus() which might result in a deadlock with kthread_park().

kthread_park() is called by the migration thread running the workqueue_online_cpu() callback.

kthread_parkme() is called by the worker thread, after it completes a work and has no locks held from that context.

reviewer
#18 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 10 lines · Peter Zijlstra
Quoted 7 lines · Lai Jiangshan
Quoted 2 lines · Peter Zijlstra

It is also the exact sequence normal per-cpu threads (smpboot) use to preserve affinity.

Other per-cpu threads normally do short-live works. wq's work can be lengthy, cpu-intensive, heavy-lock-acquiring or even call get_online_cpus() which might result in a deadlock with kthread_park().

Quoted 7 lines

kthread_park() is called by the migration thread running the workqueue_online_cpu() callback.

kthread_parkme() is called by the worker thread, after it completes a work and has no locks held from that context.

BP: AP: worker:
cpus_write_lock()
bringup_cpu() work_item_func()
bringup_wait_for_ap get_online_cpus()
kthread_park(worker)

author maintainer
#19 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 5 lines · Lai Jiangshan

BP: AP: worker:
cpus_write_lock()
bringup_cpu() work_item_func()
bringup_wait_for_ap get_online_cpus()
kthread_park(worker)

Thanks, pictures are easier. Agreed, that a problem.

I've also found another problem I think. rescuer_thread becomes part of for_each_pool_worker() between worker_attach_to_pool() and worker_detach_from_pool(), so it would try and do kthread_park() on rescuer, when things align. And rescuer_thread() doesn't have a kthread_parkme().

And we already rely on this 'ugly' thing of first doing kthread_set_per_cpu() and fixing up the affinity later for the rescuer.

Let me restart the SRCU-P testing with the below delta applied.

---
 kernel/workqueue.c | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
kernel/workqueue.c +5 -9
index 1db769b116a1..894bb885b40b 100644
@@ -2368,7 +2368,6 @@ static int worker_thread(void *__worker)
 	/* tell the scheduler that this is a workqueue worker */
 	set_pf_worker(true);
 woke_up:
-	kthread_parkme();
 	raw_spin_lock_irq(&pool->lock);
 
 	/* am I supposed to die? */
@@ -2426,7 +2425,7 @@ static int worker_thread(void *__worker)
 			move_linked_works(work, &worker->scheduled, NULL);
 			process_scheduled_works(worker);
 		}
-	} while (keep_working(pool) && !kthread_should_park());
+	} while (keep_working(pool));
 
 	worker_set_flags(worker, WORKER_PREP);
 sleep:
@@ -2438,12 +2437,9 @@ static int worker_thread(void *__worker)
 	 * event.
 	 */
 	worker_enter_idle(worker);
-	set_current_state(TASK_IDLE);
+	__set_current_state(TASK_IDLE);
 	raw_spin_unlock_irq(&pool->lock);
-
-	if (!kthread_should_park())
-		schedule();
-
+	schedule();
 	goto woke_up;
 }
 
@@ -4979,9 +4975,9 @@ static void rebind_workers(struct worker_pool *pool)
 	 * from CPU_ONLINE, the following shouldn't fail.
 	 */
 	for_each_pool_worker(worker, pool) {
-		WARN_ON_ONCE(kthread_park(worker->task) < 0);
 		kthread_set_per_cpu(worker->task, pool->cpu);
-		kthread_unpark(worker->task);
+		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
+						  pool->attrs->cpumask) < 0);
 	}
 
 	raw_spin_lock_irq(&pool->lock);
author maintainer
#20 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

Quoted 63 lines · Peter Zijlstra
Quoted 5 lines · Lai Jiangshan

BP: AP: worker:
cpus_write_lock()
bringup_cpu() work_item_func()
bringup_wait_for_ap get_online_cpus()
kthread_park(worker)

Thanks, pictures are easier. Agreed, that a problem.

I've also found another problem I think. rescuer_thread becomes part of for_each_pool_worker() between worker_attach_to_pool() and worker_detach_from_pool(), so it would try and do kthread_park() on rescuer, when things align. And rescuer_thread() doesn't have a kthread_parkme().

And we already rely on this 'ugly' thing of first doing kthread_set_per_cpu() and fixing up the affinity later for the rescuer.

Let me restart the SRCU-P testing with the below delta applied.

---
 kernel/workqueue.c | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
kernel/workqueue.c +3 -7
index 1db769b116a1..894bb885b40b 100644
@@ -2368,7 +2368,6 @@ static int worker_thread(void *__worker)
 	/* tell the scheduler that this is a workqueue worker */
 	set_pf_worker(true);
 woke_up:
-	kthread_parkme();
 	raw_spin_lock_irq(&pool->lock);
 
 	/* am I supposed to die? */
@@ -2426,7 +2425,7 @@ static int worker_thread(void *__worker)
 			move_linked_works(work, &worker->scheduled, NULL);
 			process_scheduled_works(worker);
 		}
-	} while (keep_working(pool) && !kthread_should_park());
+	} while (keep_working(pool));
 
 	worker_set_flags(worker, WORKER_PREP);
 sleep:
@@ -2438,12 +2437,9 @@ static int worker_thread(void *__worker)
 	 * event.
 	 */
 	worker_enter_idle(worker);
-	set_current_state(TASK_IDLE);
+	__set_current_state(TASK_IDLE);
 	raw_spin_unlock_irq(&pool->lock);
-
-	if (!kthread_should_park())
-		schedule();
-
+	schedule();
 	goto woke_up;
 }
 
@@ -4979,9 +4975,9 @@ static void rebind_workers(struct worker_pool *pool)
 	 * from CPU_ONLINE, the following shouldn't fail.

kernel/workqueue.c static void rebind_workers(struct worker_pool *pool) · line 4983

Quoted 10 lines
kernel/workqueue.c +2 -2
 	 */
 	for_each_pool_worker(worker, pool) {
-		WARN_ON_ONCE(kthread_park(worker->task) < 0);
 		kthread_set_per_cpu(worker->task, pool->cpu);
-		kthread_unpark(worker->task);
+		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
+						  pool->attrs->cpumask) < 0);
 	}
 
 	raw_spin_lock_irq(&pool->lock);

In the roughly 80 instances of 18*SRCU-P since sending this, I've got one sched_cpu_dying splat about a stray kworker, so somthing isn't right.

My intention was to not think today, so I'll delay that until tomorrow.

author maintainer
#21 lore raw

Re: [PATCH 3/4] workqueue: Tag bound workers with KTHREAD_IS_PER_CPU

In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Lai Jiangshan

I feel nervous to use kthread_park() here and kthread_parkme() in worker thread. And adding kthread_should_park() to the fast path also daunt me.

Is that really such a hot path that an additional load is problematic?

I think we can remove it. It would mean the kthread_park() from the online callback will take a bit longer, as it will have to wait for all the works to complete, but that should not be a fundamental problem.

author maintainer
#22 lore raw

PATCH 4/4 sched: Fix CPU hotplug / tighten is_per_cpu_kthread()

In reply to Peter Zijlstra

Prior to commit 1cf12e08bc4d ("sched/hotplug: Consolidate task migration on CPU unplug") we'd leave any task on the dying CPU and break affinity and force them off at the very end.

This scheme had to change in order to enable migrate_disable(). One cannot wait for migrate_disable() to complete while stuck in stop_machine(). Furthermore, since we need at the very least: idle, hotplug and stop threads at any point before stop_machine, we can't break affinity and/or push those away.

Under the assumption that all per-cpu kthreads are sanely handled by CPU hotplug, the new code no long breaks affinity or migrates any of them (which then includes the critical ones above).

However, there's an important difference between per-cpu kthreads and kthreads that happen to have a single CPU affinity which is lost. The latter class very much relies on the forced affinity breaking and migration semantics previously provided.

Use the new kthread_is_per_cpu() infrastructure to tighten is_per_cpu_kthread() and fix the hot-unplug problems stemming from the change.

Fixes 1cf12e08bc4d ("sched/hotplug: Consolidate task migration on CPU unplug")
---
 kernel/sched/core.c  |    8 +++++++-
 kernel/sched/sched.h |   12 ++++++++++--
 2 files changed, 17 insertions(+), 3 deletions(-)
kernel/sched/core.c +7 -1
@@ -7276,8 +7276,14 @@ static void balance_push(struct rq *rq)
 	/*
 	 * Both the cpu-hotplug and stop task are in this case and are
 	 * required to complete the hotplug process.
+	 *
+	 * XXX: the idle task does not match is_per_cpu_kthread() due to
+	 * histerical raisins.
 	 */
-	if (is_per_cpu_kthread(push_task) || is_migration_disabled(push_task)) {
+	if (rq->idle == push_task ||
+	    is_per_cpu_kthread(push_task) ||
+	    is_migration_disabled(push_task)) {
+
 		/*
 		 * If this is the idle task on the outgoing CPU try to wake
 		 * up the hotplug control thread which might wait for the
kernel/sched/sched.h +10 -2
@@ -2692,15 +2692,23 @@ static inline void membarrier_switch_mm(
 #endif
 
 #ifdef CONFIG_SMP
+/*
+ * Match geniune per-cpu kthreads; threads that are bound to a single CPU for
+ * correctness, not kernel threads that happen to have a single CPU affinity.
+ *
+ * Such threads will have PF_NO_SETAFFINITY to ensure userspace cannot
+ * accidentally place them elsewhere -- this also filters out 'early' kthreads
+ * that have PF_KTHREAD set but do not have a struct kthread.
+ */
 static inline bool is_per_cpu_kthread(struct task_struct *p)
 {
 	if (!(p->flags & PF_KTHREAD))
 		return false;
 
-	if (p->nr_cpus_allowed != 1)
+	if (!(p->flags & PF_NO_SETAFFINITY))
 		return false;
 
-	return true;
+	return kthread_is_per_cpu(p);
 }
 #endif