BUG: workqueue lockup - SRCU schedules work on not-online CPUs during size transition

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers, work accumulates, workqueue lockup detector fires.

Before 61bbcfb50514, GFP_ATOMIC allocation went straight to SRCU_SIZE_BIG, the mask = ~0 path was never reached.

Affects systems with convert_to_big active (auto when nr_cpu_ids >= 128) and possible CPUs > online CPUs. Hit on s390 LPAR (76 online, 400 possible), where possible CPUs > online CPUs is the usual case.
Also reproducible on x86 KVM --smp 16,maxcpus=255 (CONFIG_NR_CPUS=256) or simply -smp 1,maxcpus=2 with srcutree.convert_to_big=1
or --smp 16,maxcpus=64 with srcutree.big_cpu_lim=32 (CONFIG_NR_CPUS=64)

s390 log (76 online CPUs, 400 possible, all pools 76-399 stuck):

  BUG: workqueue lockup - pool cpus=76 node=0 flags=0x4 nice=0 stuck for 1842s!
  BUG: workqueue lockup - pool cpus=77 node=0 flags=0x4 nice=0 stuck for 1842s!
  ...
  BUG: workqueue lockup - pool cpus=399 node=0 flags=0x4 nice=0 stuck for 1842s!
  Showing busy workqueues and worker pools:
  workqueue rcu_gp: flags=0x108
    pwq 306: cpus=76 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    pwq 310: cpus=77 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    ...
    pwq 1598: cpus=399 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks

Not sure if replacing mask = ~0 with something derived from cpu_online_mask would be racy in that context.

[1] https://lore.kernel.org/rcu/[email protected]
[2] https://lore.kernel.org/rcu/fe28d664-3872-40f6-83c6-818627ad5b7d@paulmck-laptop

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

work accumulates, workqueue lockup detector fires.

Before 61bbcfb50514, GFP_ATOMIC allocation went straight to SRCU_SIZE_BIG, the mask = ~0 path was never reached.

Affects systems with convert_to_big active (auto when nr_cpu_ids >= 128) and possible CPUs > online CPUs. Hit on s390 LPAR (76 online, 400 possible), where possible CPUs > online CPUs is the usual case.
Also reproducible on x86 KVM --smp 16,maxcpus=255 (CONFIG_NR_CPUS=256) or simply -smp 1,maxcpus=2 with srcutree.convert_to_big=1
or --smp 16,maxcpus=64 with srcutree.big_cpu_lim=32 (CONFIG_NR_CPUS=64)

s390 log (76 online CPUs, 400 possible, all pools 76-399 stuck):

  BUG: workqueue lockup - pool cpus=76 node=0 flags=0x4 nice=0 stuck for 1842s!
  BUG: workqueue lockup - pool cpus=77 node=0 flags=0x4 nice=0 stuck for 1842s!
  ...
  BUG: workqueue lockup - pool cpus=399 node=0 flags=0x4 nice=0 stuck for 1842s!
  Showing busy workqueues and worker pools:
  workqueue rcu_gp: flags=0x108
    pwq 306: cpus=76 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    pwq 310: cpus=77 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    ...
    pwq 1598: cpus=399 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks

Not sure if replacing mask = ~0 with something derived from cpu_online_mask would be racy in that context.

[1] https://lore.kernel.org/rcu/[email protected]
[2] https://lore.kernel.org/rcu/fe28d664-3872-40f6-83c6-818627ad5b7d@paulmck-laptop

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Regards,
Boqun

Quoted 33 lines

work accumulates, workqueue lockup detector fires.

Before 61bbcfb50514, GFP_ATOMIC allocation went straight to SRCU_SIZE_BIG, the mask = ~0 path was never reached.

Affects systems with convert_to_big active (auto when nr_cpu_ids >= 128) and possible CPUs > online CPUs. Hit on s390 LPAR (76 online, 400 possible), where possible CPUs > online CPUs is the usual case.
Also reproducible on x86 KVM --smp 16,maxcpus=255 (CONFIG_NR_CPUS=256) or simply -smp 1,maxcpus=2 with srcutree.convert_to_big=1
or --smp 16,maxcpus=64 with srcutree.big_cpu_lim=32 (CONFIG_NR_CPUS=64)

s390 log (76 online CPUs, 400 possible, all pools 76-399 stuck):

  BUG: workqueue lockup - pool cpus=76 node=0 flags=0x4 nice=0 stuck for 1842s!
  BUG: workqueue lockup - pool cpus=77 node=0 flags=0x4 nice=0 stuck for 1842s!
  ...
  BUG: workqueue lockup - pool cpus=399 node=0 flags=0x4 nice=0 stuck for 1842s!
  Showing busy workqueues and worker pools:
  workqueue rcu_gp: flags=0x108
    pwq 306: cpus=76 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    pwq 310: cpus=77 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    ...
    pwq 1598: cpus=399 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks

Not sure if replacing mask = ~0 with something derived from cpu_online_mask would be racy in that context.

[1] https://lore.kernel.org/rcu/[email protected]
[2] https://lore.kernel.org/rcu/fe28d664-3872-40f6-83c6-818627ad5b7d@paulmck-laptop

Quoted 9 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

Quoted 5 lines

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Quoted 10 lines · Boqun Feng

Quoted 9 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

Quoted 8 lines

Quoted 5 lines

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We probably need to do some of the hot unplug operation during init if the CPU is possible but not online. That said, what kind of machine is it? Is the firmware just reporting bogus possible mask? How come the CPUs weren't online during boot?

Quoted 20 lines · Tejun Heo

Quoted 18 lines · Boqun Feng

Quoted 9 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

Quoted 5 lines

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Quoted 6 lines

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We probably need to do some of the hot unplug operation during init if the CPU is possible but not online. That said, what kind of machine is it? Is the firmware just reporting bogus possible mask? How come the CPUs weren't online during boot?

Just saw ibm on the cc list. Guess this was on s390?

Quoted 28 lines · Tejun Heo

Quoted 26 lines · Tejun Heo

Quoted 10 lines · Boqun Feng

Quoted 9 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

Quoted 8 lines

Quoted 5 lines

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We probably need to do some of the hot unplug operation during init if the CPU is possible but not online. That said, what kind of machine is it? Is the firmware just reporting bogus possible mask? How come the CPUs weren't online during boot?

Quoted 1 line

Just saw ibm on the cc list. Guess this was on s390?

It was indeed. What workqueue tricks does s390 play?

Quoted 16 lines · Boqun Feng

Quoted 14 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Quoted 2 lines

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We

probably need to do some of the hot unplug operation during init if the CPU

is possible but not online. That said, what kind of machine is it? Is the firmware just reporting bogus possible mask? How come the CPUs weren't online during boot?

Thanks.

Signature · 1 line

tejun

Quoted 29 lines · Paul E. McKenney

Quoted 28 lines · Tejun Heo

Quoted 26 lines · Tejun Heo

Quoted 10 lines · Boqun Feng

Quoted 9 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

Quoted 8 lines

Quoted 5 lines

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We probably need to do some of the hot unplug operation during init if the CPU is possible but not online. That said, what kind of machine is it? Is the firmware just reporting bogus possible mask? How come the CPUs weren't online during boot?

Quoted 3 lines

Quoted 1 line

Just saw ibm on the cc list. Guess this was on s390?

It was indeed. What workqueue tricks does s390 play?

They just come up with genuinely possible but offline CPUs. Most setups don't do that. I'll spin up a patch later today.

Quoted 17 lines · Tejun Heo

Quoted 16 lines · Boqun Feng

Quoted 14 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Quoted 5 lines

Quoted 2 lines

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We

;-) ;-) ;-)

Quoted 1 line

probably need to do some of the hot unplug operation during init if the CPU

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

[1]: https://lore.kernel.org/rcu/073abb55-197a-4519-b177-f9f776624fed@paulmck-laptop/

Regards,
Boqun

Quoted 8 lines

is possible but not online. That said, what kind of machine is it? Is the firmware just reporting bogus possible mask? How come the CPUs weren't online during boot?

Thanks.

Signature · 1 line

tejun

Quoted 35 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers, work accumulates, workqueue lockup detector fires.

Before 61bbcfb50514, GFP_ATOMIC allocation went straight to SRCU_SIZE_BIG, the mask = ~0 path was never reached.

Affects systems with convert_to_big active (auto when nr_cpu_ids >= 128) and possible CPUs > online CPUs. Hit on s390 LPAR (76 online, 400 possible), where possible CPUs > online CPUs is the usual case.
Also reproducible on x86 KVM --smp 16,maxcpus=255 (CONFIG_NR_CPUS=256) or simply -smp 1,maxcpus=2 with srcutree.convert_to_big=1
or --smp 16,maxcpus=64 with srcutree.big_cpu_lim=32 (CONFIG_NR_CPUS=64)

s390 log (76 online CPUs, 400 possible, all pools 76-399 stuck):

  BUG: workqueue lockup - pool cpus=76 node=0 flags=0x4 nice=0 stuck for 1842s!
  BUG: workqueue lockup - pool cpus=77 node=0 flags=0x4 nice=0 stuck for 1842s!
  ...
  BUG: workqueue lockup - pool cpus=399 node=0 flags=0x4 nice=0 stuck for 1842s!
  Showing busy workqueues and worker pools:
  workqueue rcu_gp: flags=0x108
    pwq 306: cpus=76 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    pwq 310: cpus=77 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    ...
    pwq 1598: cpus=399 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks

Quoted 5 lines

Not sure if replacing mask = ~0 with something derived from cpu_online_mask would be racy in that context.

[1] https://lore.kernel.org/rcu/[email protected]
[2] https://lore.kernel.org/rcu/fe28d664-3872-40f6-83c6-818627ad5b7d@paulmck-laptop

This was a pre-existing bug, but the change made it much more likely to happen.

Does the alleged (and untested) fix below do the trick? The theory is that if a given CPU has ever been fully online, it has workqueues set up. Directly checking whether a CPU is currently online is vulnerable to a CPU piling up lots of SRCU callbacks, then going offline. So we do need to be prepared to invoke SRCU callbacks for CPUs that are currently offline.

kernel/rcu/srcutree.c +1 -1

index 0d01cd8c4b4a7..e68ee7f69e1fc 100644
@@ -898,7 +898,7 @@ static void srcu_schedule_cbs_snp(struct srcu_struct *ssp, struct srcu_node *snp
 	int cpu;
 
 	for (cpu = snp->grplo; cpu <= snp->grphi; cpu++) {
-		if (!(mask & (1UL << (cpu - snp->grplo))))
+		if (!(mask & (1UL << (cpu - snp->grplo))) || !rcu_cpu_beenfullyonline(cpu))
 			continue;
 		srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, cpu), delay);
 	}

Quoted 36 lines · Paul E. McKenney

Quoted 35 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers, work accumulates, workqueue lockup detector fires.

Before 61bbcfb50514, GFP_ATOMIC allocation went straight to SRCU_SIZE_BIG, the mask = ~0 path was never reached.

Affects systems with convert_to_big active (auto when nr_cpu_ids >= 128) and possible CPUs > online CPUs. Hit on s390 LPAR (76 online, 400 possible), where possible CPUs > online CPUs is the usual case.
Also reproducible on x86 KVM --smp 16,maxcpus=255 (CONFIG_NR_CPUS=256) or simply -smp 1,maxcpus=2 with srcutree.convert_to_big=1
or --smp 16,maxcpus=64 with srcutree.big_cpu_lim=32 (CONFIG_NR_CPUS=64)

s390 log (76 online CPUs, 400 possible, all pools 76-399 stuck):

  BUG: workqueue lockup - pool cpus=76 node=0 flags=0x4 nice=0 stuck for 1842s!
  BUG: workqueue lockup - pool cpus=77 node=0 flags=0x4 nice=0 stuck for 1842s!
  ...
  BUG: workqueue lockup - pool cpus=399 node=0 flags=0x4 nice=0 stuck for 1842s!
  Showing busy workqueues and worker pools:
  workqueue rcu_gp: flags=0x108
    pwq 306: cpus=76 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    pwq 310: cpus=77 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks
    ...
    pwq 1598: cpus=399 node=0 flags=0x4 nice=0 active=3 refcnt=4
      pending: 3*srcu_invoke_callbacks

Quoted 8 lines

Quoted 5 lines

Not sure if replacing mask = ~0 with something derived from cpu_online_mask would be racy in that context.

[1] https://lore.kernel.org/rcu/[email protected]
[2] https://lore.kernel.org/rcu/fe28d664-3872-40f6-83c6-818627ad5b7d@paulmck-laptop

This was a pre-existing bug, but the change made it much more likely to happen.

Yes, indeed.

Quoted 5 lines

Does the alleged (and untested) fix below do the trick? The theory is that if a given CPU has ever been fully online, it has workqueues set up. Directly checking whether a CPU is currently online is vulnerable to a CPU piling up lots of SRCU callbacks, then going offline. So we do need to be prepared to invoke SRCU callbacks for CPUs that are currently offline.

Yes, tested on s390 LPAR (76 online, 400 possible) as well as on x86 KVM with --smp 16,maxcpus=255 and CONFIG_NR_CPUS=256 no more workqueue lockup in both cases.

Thank you!

Tested-by Vasily Gorbik <[email protected]>

On Thu, Apr 09, 2026 at 09:15:50PM +0200, Vasily Gorbik wrote:

Quoted 6 lines

Yes, tested on s390 LPAR (76 online, 400 possible) as well as on x86 KVM with --smp 16,maxcpus=255 and CONFIG_NR_CPUS=256 no more workqueue lockup in both cases.

Thank you!

Quoted 1 line

Tested-by Vasily Gorbik <[email protected]>

Thank you for testing this!

Please see below for an updated patch. Tejun's patch might obsolete this one, but just in case he balks at SRCU queueing handlers for CPUs that are not even in the cpu_possible_mask. ;-)

Quoted 17 lines · Tejun Heo

Quoted 16 lines · Boqun Feng

Quoted 14 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Quoted 5 lines

Quoted 2 lines

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We

;-) ;-) ;-)

Quoted 1 line

probably need to do some of the hot unplug operation during init if the CPU

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

Hello,

Quoted 26 lines · Boqun Feng

Quoted 17 lines · Tejun Heo

Quoted 16 lines · Boqun Feng

Quoted 14 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

[Cc workqueue]

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Quoted 5 lines

Quoted 2 lines

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We

;-) ;-) ;-)

Quoted 8 lines

Quoted 1 line

probably need to do some of the hot unplug operation during init if the CPU

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

The easiest way to do this is just creating the initial workers for all possible pools. Please see below. However, the downside is that it's going to create all workers for all possible cpus. This isn't a problem for anybody else but these IBM mainframes often come up with a lot of possible but not-yet-or-ever-online CPUs for capacity management, so the cost may not be negligible on some configurations.

IBM folks, is that okay?

Also, why do you need to queue work items on an offline CPU? Do they actually have to be per-cpu? Can you get away with using an unbound workqueue?

Thanks.

From: Tejun Heo <[email protected]>
Subject: workqueue: Create workers for all possible CPUs on init

Per-CPU worker pools are initialized for every possible CPU during early boot, but workqueue_init() only creates initial workers for online CPUs. On systems where possible CPUs outnumber online CPUs (e.g. s390 LPARs with 76 online and 400 possible CPUs), the pools for never-onlined CPUs have POOL_DISASSOCIATED set but no workers. Any work item queued on such a CPU hangs indefinitely.

This was exposed by 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") which made SRCU schedule callbacks on all possible CPUs during size transitions, triggering workqueue lockup warnings for all never-onlined CPUs.

Create workers for all possible CPUs during init, not just online ones. For online CPUs, the behavior is unchanged - POOL_DISASSOCIATED is cleared and the worker is bound to the CPU. For not-yet-online CPUs, POOL_DISASSOCIATED remains set, so worker_attach_to_pool() marks the worker UNBOUND and it can execute on any CPU. When the CPU later comes online, rebind_workers() handles the transition to associated operation as usual.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Tejun Heo <[email protected]>

Cc Boqun Feng <[email protected]>

Cc Paul E. McKenney <[email protected]>

---
 kernel/workqueue.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

kernel/workqueue.c +3 -2

@@ -8068,9 +8068,10 @@ void __init workqueue_init(void)
 		for_each_bh_worker_pool(pool, cpu)
 			BUG_ON(!create_worker(pool));
 
-	for_each_online_cpu(cpu) {
+	for_each_possible_cpu(cpu) {
 		for_each_cpu_worker_pool(pool, cpu) {
-			pool->flags &= ~POOL_DISASSOCIATED;
+			if (cpu_online(cpu))
+				pool->flags &= ~POOL_DISASSOCIATED;
 			BUG_ON(!create_worker(pool));
 		}
 	}

Signature · 1 line

tejun

Quoted 8 lines

The easiest way to do this is just creating the initial workers for all possible pools. Please see below. However, the downside is that it's going to create all workers for all possible cpus. This isn't a problem for anybody else but these IBM mainframes often come up with a lot of possible but not-yet-or-ever-online CPUs for capacity management, so the cost may not be negligible on some configurations.

IBM folks, is that okay?

I have also seen x86 systems whose firmware claimed very large numbers of CPUs. :-(

Quoted 3 lines

Also, why do you need to queue work items on an offline CPU? Do they actually have to be per-cpu? Can you get away with using an unbound workqueue?

It is good for them to run on the specified CPU in the common case for cache-locality reasons, but if they were occasionally redirected to some other CPU, that would be just fine.

I am also keeping the patch that avoids queueing work to CPUs that are not yet fully online. Further adjustments will be needed if someone invokes call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() from an CPU that is not yet fully online. Past experience of course suggests that this will be happen, and that there will be a good reason for it. ;-)

Quoted 1 line · Paul E. McKenney

On Thu, Apr 09, 2026 at 09:15:50PM +0200, Vasily Gorbik wrote:

[ . . . ]

Quoted 6 lines

Quoted 6 lines

Yes, tested on s390 LPAR (76 online, 400 possible) as well as on x86 KVM with --smp 16,maxcpus=255 and CONFIG_NR_CPUS=256 no more workqueue lockup in both cases.

Thank you!

Quoted 7 lines

Quoted 1 line

Tested-by Vasily Gorbik <[email protected]>

Thank you for testing this!

Please see below for an updated patch. Tejun's patch might obsolete this one, but just in case he balks at SRCU queueing handlers for CPUs that are not even in the cpu_possible_mask. ;-)

And because we don't invoke SRCU callbacks on CPUs that are not yet fully online, such CPUs had better not invoke call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() on a CPU that is not yet fully online. I am therefore adding the warning shown below.

Better paranoid late than paranoid not at all. ;-)

Hello,

Quoted 6 lines

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

The easiest way to do this is just creating the initial workers for all possible pools. Please see below. However, the downside is that it's going to create all workers for all possible cpus. This isn't a problem for anybody else but these IBM mainframes often come up with a lot of possible but not-yet-or-ever-online CPUs for capacity management, so the cost may not be negligible on some configurations.

IBM folks, is that okay?

Also, why do you need to queue work items on an offline CPU? Do they actually have to be per-cpu? Can you get away with using an unbound workqueue?

Thanks.

From: Tejun Heo <[email protected]>
Subject: workqueue: Create workers for all possible CPUs on init

Per-CPU worker pools are initialized for every possible CPU during early boot, but workqueue_init() only creates initial workers for online CPUs. On systems where possible CPUs outnumber online CPUs (e.g. s390 LPARs with 76 online and 400 possible CPUs), the pools for never-onlined CPUs have POOL_DISASSOCIATED set but no workers. Any work item queued on such a CPU hangs indefinitely.

This was exposed by 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") which made SRCU schedule callbacks on all possible CPUs during size transitions, triggering workqueue lockup warnings for all never-onlined CPUs.

Create workers for all possible CPUs during init, not just online ones. For online CPUs, the behavior is unchanged - POOL_DISASSOCIATED is cleared and the worker is bound to the CPU. For not-yet-online CPUs, POOL_DISASSOCIATED remains set, so worker_attach_to_pool() marks the worker UNBOUND and it can execute on any CPU. When the CPU later comes online, rebind_workers() handles the transition to associated operation as usual.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Tejun Heo <[email protected]>

Cc Boqun Feng <[email protected]>

Cc Paul E. McKenney <[email protected]>

---
 kernel/workqueue.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

kernel/workqueue.c +3 -2

@@ -8068,9 +8068,10 @@ void __init workqueue_init(void)
 		for_each_bh_worker_pool(pool, cpu)
 			BUG_ON(!create_worker(pool));
 
-	for_each_online_cpu(cpu) {
+	for_each_possible_cpu(cpu) {
 		for_each_cpu_worker_pool(pool, cpu) {
-			pool->flags &= ~POOL_DISASSOCIATED;
+			if (cpu_online(cpu))
+				pool->flags &= ~POOL_DISASSOCIATED;
 			BUG_ON(!create_worker(pool));
 		}
 	}

Signature · 1 line

tejun

Quoted 9 lines · Tejun Heo

Hello,

Quoted 6 lines

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

Quoted 8 lines

The easiest way to do this is just creating the initial workers for all possible pools. Please see below. However, the downside is that it's going to create all workers for all possible cpus. This isn't a problem for anybody else but these IBM mainframes often come up with a lot of possible but not-yet-or-ever-online CPUs for capacity management, so the cost may not be negligible on some configurations.

IBM folks, is that okay?

Even on PowerPC LPARS, its not uncommon to have possible cpus != online cpus at boot. However your approach will work.

And Samir has already tested the same too and reported here
https://lkml.kernel.org/r/[email protected]

Quoted 14 lines

From: Tejun Heo <[email protected]>
Subject: workqueue: Create workers for all possible CPUs on init

Per-CPU worker pools are initialized for every possible CPU during early boot, but workqueue_init() only creates initial workers for online CPUs. On systems where possible CPUs outnumber online CPUs (e.g. s390 LPARs with 76 online and 400 possible CPUs), the pools for never-onlined CPUs have POOL_DISASSOCIATED set but no workers. Any work item queued on such a CPU hangs indefinitely.

This was exposed by 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") which made SRCU schedule callbacks on all possible CPUs during size transitions, triggering workqueue lockup warnings for all never-onlined CPUs.

Quoted 7 lines

Create workers for all possible CPUs during init, not just online ones. For online CPUs, the behavior is unchanged - POOL_DISASSOCIATED is cleared and the worker is bound to the CPU. For not-yet-online CPUs, POOL_DISASSOCIATED remains set, so worker_attach_to_pool() marks the worker UNBOUND and it can execute on any CPU. When the CPU later comes online, rebind_workers() handles the transition to associated operation as usual.

With these patch, if a CPU has been onlined once, it's should be ok to queue the work on that CPU even if its offline now.

Quoted 47 lines · Srikar Dronamraju

Quoted 9 lines · Tejun Heo

Hello,

Quoted 6 lines

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

Quoted 8 lines

The easiest way to do this is just creating the initial workers for all possible pools. Please see below. However, the downside is that it's going to create all workers for all possible cpus. This isn't a problem for anybody else but these IBM mainframes often come up with a lot of possible but not-yet-or-ever-online CPUs for capacity management, so the cost may not be negligible on some configurations.

IBM folks, is that okay?

Even on PowerPC LPARS, its not uncommon to have possible cpus != online cpus at boot. However your approach will work.

And Samir has already tested the same too and reported here
https://lkml.kernel.org/r/[email protected]

Quoted 14 lines

From: Tejun Heo <[email protected]>
Subject: workqueue: Create workers for all possible CPUs on init

Per-CPU worker pools are initialized for every possible CPU during early boot, but workqueue_init() only creates initial workers for online CPUs. On systems where possible CPUs outnumber online CPUs (e.g. s390 LPARs with 76 online and 400 possible CPUs), the pools for never-onlined CPUs have POOL_DISASSOCIATED set but no workers. Any work item queued on such a CPU hangs indefinitely.

This was exposed by 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") which made SRCU schedule callbacks on all possible CPUs during size transitions, triggering workqueue lockup warnings for all never-onlined CPUs.

Quoted 7 lines

Create workers for all possible CPUs during init, not just online ones. For online CPUs, the behavior is unchanged - POOL_DISASSOCIATED is cleared and the worker is bound to the CPU. For not-yet-online CPUs, POOL_DISASSOCIATED remains set, so worker_attach_to_pool() marks the worker UNBOUND and it can execute on any CPU. When the CPU later comes online, rebind_workers() handles the transition to associated operation as usual.

Quoted 2 lines

With these patch, if a CPU has been onlined once, it's should be ok to queue the work on that CPU even if its offline now.

That already seems to hold without this patch, what this patch newly covers is queueing on CPUs that have never been online.

Do we actually need to create workers for every possible CPU at boot? On the s390 LPAR in question (76 online / 400 possible) that's a few hundred extra kthreads kept around for the life of the system. That's probably the same on PowerPC.

Wouldn't Paul's SRCU-side fix [1] alone be enough here for PowerPC as well? I retested it on s390 (76/400) and on x86 KVM with --smp 16,maxcpus=255 and the lockup didn't reproduce in either case.

[1] https://lore.kernel.org/rcu/ed1fa6cd-7343-4ca3-8b9d-d699ca496f83@paulmck-laptop/

Quoted 63 lines · Vasily Gorbik

Quoted 47 lines · Srikar Dronamraju

Quoted 9 lines · Tejun Heo

Hello,

Quoted 6 lines

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

Quoted 8 lines

The easiest way to do this is just creating the initial workers for all possible pools. Please see below. However, the downside is that it's going to create all workers for all possible cpus. This isn't a problem for anybody else but these IBM mainframes often come up with a lot of possible but not-yet-or-ever-online CPUs for capacity management, so the cost may not be negligible on some configurations.

IBM folks, is that okay?

Even on PowerPC LPARS, its not uncommon to have possible cpus != online cpus at boot. However your approach will work.

And Samir has already tested the same too and reported here
https://lkml.kernel.org/r/[email protected]

Quoted 14 lines

From: Tejun Heo <[email protected]>
Subject: workqueue: Create workers for all possible CPUs on init

Per-CPU worker pools are initialized for every possible CPU during early boot, but workqueue_init() only creates initial workers for online CPUs. On systems where possible CPUs outnumber online CPUs (e.g. s390 LPARs with 76 online and 400 possible CPUs), the pools for never-onlined CPUs have POOL_DISASSOCIATED set but no workers. Any work item queued on such a CPU hangs indefinitely.

This was exposed by 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") which made SRCU schedule callbacks on all possible CPUs during size transitions, triggering workqueue lockup warnings for all never-onlined CPUs.

Quoted 7 lines

Create workers for all possible CPUs during init, not just online ones. For online CPUs, the behavior is unchanged - POOL_DISASSOCIATED is cleared and the worker is bound to the CPU. For not-yet-online CPUs, POOL_DISASSOCIATED remains set, so worker_attach_to_pool() marks the worker UNBOUND and it can execute on any CPU. When the CPU later comes online, rebind_workers() handles the transition to associated operation as usual.

Quoted 2 lines

With these patch, if a CPU has been onlined once, it's should be ok to queue the work on that CPU even if its offline now.

That already seems to hold without this patch, what this patch newly covers is queueing on CPUs that have never been online.

Do we actually need to create workers for every possible CPU at boot? On the s390 LPAR in question (76 online / 400 possible) that's a few hundred extra kthreads kept around for the life of the system. That's probably the same on PowerPC.

Wouldn't Paul's SRCU-side fix [1] alone be enough here for PowerPC as well? I retested it on s390 (76/400) and on x86 KVM with --smp 16,maxcpus=255 and the lockup didn't reproduce in either case.

Quoted 1 line

[1] https://lore.kernel.org/rcu/ed1fa6cd-7343-4ca3-8b9d-d699ca496f83@paulmck-laptop/

Just to emphasize that SRCU really was buggy before my fix. The queue_work_on() kernel-doc header clearly states the rules. The bug is even more embarrassing given just who it was that wrote those two sentences. ;-)

Thanx, Paul

/**
* queue_work_on - queue work on specific cpu
* @cpu: CPU number to execute work on
* @wq: workqueue to use
* @work: work to queue
*
* We queue the work to a specific CPU, the caller must ensure it
* can't go away. Callers that fail to ensure that the specified
* CPU cannot go away will execute on a randomly chosen CPU.
* But note well that callers specifying a CPU that never has been
* online will get a splat.
*
* Return: %false if @work was already on a queue, %true otherwise. */

On Thu, Apr 09, 2026 at 09:03:26PM -0700, Paul E. McKenney wrote:
Please see below for the full patch, including refraining from queueing workqueue handlers on not-yet-online CPUs and diverting SRCU callbacks from not-yet-fully-online CPUs to the boot CPU's callback queue.

commit ce533a60b2ef29a9b516cc717e77c6b679bc09c0

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 9 11:16:02 2026 -0700

srcu: Don't queue workqueue handlers to never-online CPUs

While an srcu_struct structure is in the midst of switching from CPU-0 to all-CPUs state, it can attempt to invoke callbacks for CPUs that have never been online. Worse yet, it can attempt in invoke callbacks for CPUs that never will be online due to not being present in the cpu_possible_mask. This can cause hangs on s390, which is not set up to deal with workqueue handlers being scheduled on such CPUs. This commit therefore causes Tree SRCU to refrain from queueing workqueue handlers on CPUs that have not yet (and might never) come online.

Because callbacks are not invoked on CPUs that have not been online, it is an error to invoke call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() on a CPU that is not yet fully online. However, it turns out to be less code to redirect the callbacks from too-early invocations of call_srcu() than to warn about such invocations. This commit therefore also redirects callbacks queued on not-yet-fully-online CPUs to the boot CPU.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Tested-by Vasily Gorbik <[email protected]>

Cc Tejun Heo <[email protected]>

I have limited understanding in rcu or workqueues, but my two cents.

Quoted 64 lines · Paul E. McKenney

Quoted 63 lines · Vasily Gorbik

Quoted 47 lines · Srikar Dronamraju

Quoted 9 lines · Tejun Heo

Hello,

Quoted 6 lines

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

Quoted 8 lines

The easiest way to do this is just creating the initial workers for all possible pools. Please see below. However, the downside is that it's going to create all workers for all possible cpus. This isn't a problem for anybody else but these IBM mainframes often come up with a lot of possible but not-yet-or-ever-online CPUs for capacity management, so the cost may not be negligible on some configurations.

IBM folks, is that okay?

Even on PowerPC LPARS, its not uncommon to have possible cpus != online cpus at boot. However your approach will work.

And Samir has already tested the same too and reported here
https://lkml.kernel.org/r/[email protected]

Quoted 14 lines

From: Tejun Heo <[email protected]>
Subject: workqueue: Create workers for all possible CPUs on init

Per-CPU worker pools are initialized for every possible CPU during early boot, but workqueue_init() only creates initial workers for online CPUs. On systems where possible CPUs outnumber online CPUs (e.g. s390 LPARs with 76 online and 400 possible CPUs), the pools for never-onlined CPUs have POOL_DISASSOCIATED set but no workers. Any work item queued on such a CPU hangs indefinitely.

This was exposed by 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") which made SRCU schedule callbacks on all possible CPUs during size transitions, triggering workqueue lockup warnings for all never-onlined CPUs.

Quoted 7 lines

Create workers for all possible CPUs during init, not just online ones. For online CPUs, the behavior is unchanged - POOL_DISASSOCIATED is cleared and the worker is bound to the CPU. For not-yet-online CPUs, POOL_DISASSOCIATED remains set, so worker_attach_to_pool() marks the worker UNBOUND and it can execute on any CPU. When the CPU later comes online, rebind_workers() handles the transition to associated operation as usual.

Quoted 2 lines

With these patch, if a CPU has been onlined once, it's should be ok to queue the work on that CPU even if its offline now.

That already seems to hold without this patch, what this patch newly covers is queueing on CPUs that have never been online.

Do we actually need to create workers for every possible CPU at boot? On the s390 LPAR in question (76 online / 400 possible) that's a few hundred extra kthreads kept around for the life of the system. That's probably the same on PowerPC.

Wouldn't Paul's SRCU-side fix [1] alone be enough here for PowerPC as well? I retested it on s390 (76/400) and on x86 KVM with --smp 16,maxcpus=255 and the lockup didn't reproduce in either case.

Quoted 6 lines

Quoted 1 line

[1] https://lore.kernel.org/rcu/ed1fa6cd-7343-4ca3-8b9d-d699ca496f83@paulmck-laptop/

Just to emphasize that SRCU really was buggy before my fix. The queue_work_on() kernel-doc header clearly states the rules. The bug is even more embarrassing given just who it was that wrote those two sentences. ;-)

That mask = ~0 is really looks uncomfortable to me. What does it mean? It might end up even sending to non possible CPUs without proper checks.

It should use either cpumask_setall? or use cpu_online_mask?

Your current patch rcu_cpu_beenfullyonline indicates that code around srcu_schedule_cbs_sdp handles hotplug already right?
in that case, just setting mask = cpu_online_mask would work?

Quoted 6 lines

/**
* queue_work_on - queue work on specific cpu
* @cpu: CPU number to execute work on
* @wq: workqueue to use
* @work: work to queue
*

Quoted 8 lines

* We queue the work to a specific CPU, the caller must ensure it
* can't go away. Callers that fail to ensure that the specified
* CPU cannot go away will execute on a randomly chosen CPU.
* But note well that callers specifying a CPU that never has been
* online will get a splat.
*
* Return: %false if @work was already on a queue, %true otherwise. */

In that case, making offline CPUs have a unbound workqueue is wrong. no?

It might encourage more users to abuse queue_work_on interface to send to offline CPUs without any checks and onus now falls onto workqueue to disaptch to unbound wqs.

So I think it is better to put the guardrails in SRCU instead of any change in workqueue.

Quoted 4 lines · Paul E. McKenney

On Thu, Apr 09, 2026 at 09:03:26PM -0700, Paul E. McKenney wrote:
Please see below for the full patch, including refraining from queueing workqueue handlers on not-yet-online CPUs and diverting SRCU callbacks from not-yet-fully-online CPUs to the boot CPU's callback queue.

...

Quoted 23 lines

commit ce533a60b2ef29a9b516cc717e77c6b679bc09c0

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 9 11:16:02 2026 -0700

srcu: Don't queue workqueue handlers to never-online CPUs

While an srcu_struct structure is in the midst of switching from CPU-0 to all-CPUs state, it can attempt to invoke callbacks for CPUs that have never been online. Worse yet, it can attempt in invoke callbacks for CPUs that never will be online due to not being present in the cpu_possible_mask. This can cause hangs on s390, which is not set up to deal with workqueue handlers being scheduled on such CPUs. This commit therefore causes Tree SRCU to refrain from queueing workqueue handlers on CPUs that have not yet (and might never) come online.

Because callbacks are not invoked on CPUs that have not been online, it is an error to invoke call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() on a CPU that is not yet fully online. However, it turns out to be less code to redirect the callbacks from too-early invocations of call_srcu() than to warn about such invocations. This commit therefore also redirects callbacks queued on not-yet-fully-online CPUs to the boot CPU.

Quoted 4 lines

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Tested-by Vasily Gorbik <[email protected]>

Cc Tejun Heo <[email protected]>

I retested it on s390 and on x86 KVM with --smp 16,maxcpus=255, all looks good to me.

FWIW, again:

Tested-by Vasily Gorbik <[email protected]>

Would you mind adding Cc: stable so it gets picked up for v7.0? 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") is what made it reproducible for us.

Thank you!

Hello,

Quoted 22 lines · Boqun Feng

Quoted 12 lines · Tejun Heo

Quoted 11 lines · Boqun Feng

Quoted 10 lines · Boqun Feng

Quoted 7 lines · Vasily Gorbik

Commit 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") defers srcu_node tree allocation when called under raw spinlock, putting SRCU through ~6 transitional grace periods (SRCU_SIZE_ALLOC to SRCU_SIZE_BIG). During this transition srcu_gp_end() uses mask = ~0, which makes srcu_schedule_cbs_snp() call queue_work_on() for every possible CPU. Since rcu_gp_wq is WQ_PERCPU, work targets per-CPU pools directly - pools for not-online CPUs have no workers,

[Cc workqueue]

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

Hmm.. I thought for offline CPUs the corresponding worker pools become a unbound one hence there are still workers?

Ah, as Paul replied in another email, the problem was because these CPUs had never been onlined, so they don't even have unbound workers?

Hahaha, we do initialize worker pool for every possible CPU but the transition to unbound operation happens in the hot unplug callback. We

;-) ;-) ;-)

Quoted 7 lines

Quoted 1 line

probably need to do some of the hot unplug operation during init if the CPU

Seems that we (mostly Paul) have our own trick to track whether a CPU has ever been onlined in RCU, see rcu_cpu_beenfullyonline(). Paul also used it in his fix [1]. And I think it won't be that hard to copy it into workqueue and let queue_work_on() use it so that if the user queues a work on a never-onlined CPU, it can detect it (with a warning?) and do something?

The easiest way to do this is just creating the initial workers for all possible pools. Please see below. However, the downside is that it's going to create all workers for all possible cpus. This isn't a problem for anybody else but these IBM mainframes often come up with a lot of possible but not-yet-or-ever-online CPUs for capacity management, so the cost may not be negligible on some configurations.

IBM folks, is that okay?

Also, why do you need to queue work items on an offline CPU? Do they actually have to be per-cpu? Can you get away with using an unbound workqueue?

Thanks.

From: Tejun Heo <[email protected]>
Subject: workqueue: Create workers for all possible CPUs on init

Per-CPU worker pools are initialized for every possible CPU during early boot, but workqueue_init() only creates initial workers for online CPUs. On systems where possible CPUs outnumber online CPUs (e.g. s390 LPARs with 76 online and 400 possible CPUs), the pools for never-onlined CPUs have POOL_DISASSOCIATED set but no workers. Any work item queued on such a CPU hangs indefinitely.

This was exposed by 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") which made SRCU schedule callbacks on all possible CPUs during size transitions, triggering workqueue lockup warnings for all never-onlined CPUs.

Create workers for all possible CPUs during init, not just online ones. For online CPUs, the behavior is unchanged - POOL_DISASSOCIATED is cleared and the worker is bound to the CPU. For not-yet-online CPUs, POOL_DISASSOCIATED remains set, so worker_attach_to_pool() marks the worker UNBOUND and it can execute on any CPU. When the CPU later comes online, rebind_workers() handles the transition to associated operation as usual.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Tejun Heo <[email protected]>

Cc Boqun Feng <[email protected]>

Cc Paul E. McKenney <[email protected]>

---
  kernel/workqueue.c |    5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)

kernel/workqueue.c +3 -2

@@ -8068,9 +8068,10 @@ void __init workqueue_init(void)
  		for_each_bh_worker_pool(pool, cpu)
  			BUG_ON(!create_worker(pool));
 
-	for_each_online_cpu(cpu) {
+	for_each_possible_cpu(cpu) {
  		for_each_cpu_worker_pool(pool, cpu) {
-			pool->flags &= ~POOL_DISASSOCIATED;
+			if (cpu_online(cpu))
+				pool->flags &= ~POOL_DISASSOCIATED;
  			BUG_ON(!create_worker(pool));
  		}
  	}

Quoted 12 lines · Vasily Gorbik

Quoted 4 lines · Paul E. McKenney

On Thu, Apr 09, 2026 at 09:03:26PM -0700, Paul E. McKenney wrote:
Please see below for the full patch, including refraining from queueing workqueue handlers on not-yet-online CPUs and diverting SRCU callbacks from not-yet-fully-online CPUs to the boot CPU's callback queue.

...

Quoted 6 lines

commit ce533a60b2ef29a9b516cc717e77c6b679bc09c0

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 9 11:16:02 2026 -0700

srcu: Don't queue workqueue handlers to never-online CPUs

Quoted 11 lines

Quoted 5 lines

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Tested-by Vasily Gorbik <[email protected]>

Cc Tejun Heo <[email protected]>

I retested it on s390 and on x86 KVM with --smp 16,maxcpus=255, all looks good to me.

FWIW, again:

Quoted 7 lines

Tested-by Vasily Gorbik <[email protected]>

Would you mind adding Cc: stable so it gets picked up for v7.0? 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible") is what made it reproducible for us.

Thank you!

And thank you for testing it, plus apologies for the hassle!

At my next rebase, I will add the following:

Fixes 61bbcfb50514 ("srcu: Push srcu_node allocation to GP when non-preemptible")

Tested-by Vasily Gorbik <[email protected]>

That should pull it into the needed -stable releases.

Seem reasonable?

Quoted 8 lines

That mask = ~0 is really looks uncomfortable to me. What does it mean? It might end up even sending to non possible CPUs without proper checks.

It should use either cpumask_setall? or use cpu_online_mask?

Your current patch rcu_cpu_beenfullyonline indicates that code around srcu_schedule_cbs_sdp handles hotplug already right?
in that case, just setting mask = cpu_online_mask would work?

Agreed. Which is why I have this commit queued:

f8d5aaaf90f8 ("srcu: Don't queue workqueue handlers to never-online CPUs")

This is currently slated for the upcoming merge window, but if you need it sooner, please let us know. Please see the end of this email for the full commit.

Thanx, Paul

Quoted 16 lines

Quoted 6 lines

/**
* queue_work_on - queue work on specific cpu
* @cpu: CPU number to execute work on
* @wq: workqueue to use
* @work: work to queue
*

Quoted 8 lines

* We queue the work to a specific CPU, the caller must ensure it
* can't go away. Callers that fail to ensure that the specified
* CPU cannot go away will execute on a randomly chosen CPU.
* But note well that callers specifying a CPU that never has been
* online will get a splat.
*
* Return: %false if @work was already on a queue, %true otherwise. */

Quoted 8 lines

In that case, making offline CPUs have a unbound workqueue is wrong. no?

It might encourage more users to abuse queue_work_on interface to send to offline CPUs without any checks and onus now falls onto workqueue to disaptch to unbound wqs.

So I think it is better to put the guardrails in SRCU instead of any change in workqueue.

------------------------------------------------------------------------

commit f8d5aaaf90f8294890802ce8dccbafd9850ac5f9

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 9 11:16:02 2026 -0700

srcu: Don't queue workqueue handlers to never-online CPUs

While an srcu_struct structure is in the midst of switching from CPU-0 to all-CPUs state, it can attempt to invoke callbacks for CPUs that have never been online. Worse yet, it can attempt in invoke callbacks for CPUs that never will be online due to not being present in the

cpu_possible_mask. This can cause hangs on s390, which is not set up to deal with workqueue handlers being scheduled on such CPUs. This commit therefore causes Tree SRCU to refrain from queueing workqueue handlers on CPUs that have not yet (and might never) come online.

Because callbacks are not invoked on CPUs that have not been online, it is an error to invoke call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() on a CPU that is not yet fully online. However, it turns out to be less code to redirect the callbacks from too-early invocations of call_srcu() than to warn about such invocations. This commit therefore also redirects callbacks queued on not-yet-fully-online CPUs to the boot CPU.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Tested-by Vasily Gorbik <[email protected]>

Cc Tejun Heo <[email protected]>

kernel/rcu/srcutree.c +3 -5

index 0d01cd8c4b4a7b..7c2f7cc131f7ae 100644
@@ -897,11 +897,9 @@ static void srcu_schedule_cbs_snp(struct srcu_struct *ssp, struct srcu_node *snp
  {
  	int cpu;
 
-	for (cpu = snp->grplo; cpu <= snp->grphi; cpu++) {
-		if (!(mask & (1UL << (cpu - snp->grplo))))
-			continue;
-		srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, cpu), delay);
-	}
+	for (cpu = snp->grplo; cpu <= snp->grphi; cpu++)
+		if ((mask & (1UL << (cpu - snp->grplo))) && rcu_cpu_beenfullyonline(cpu))
+			srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, cpu), delay);
  }
 
  /*

kernel/rcu/srcutree.c +0 -0

  		sdp = per_cpu_ptr(ssp->sda, get_boot_cpu_id());
  	else
  		sdp = raw_cpu_ptr(ssp->sda);

Hi Paul.

Quoted 48 lines · Paul E. McKenney

Quoted 8 lines

That mask = ~0 is really looks uncomfortable to me. What does it mean? It might end up even sending to non possible CPUs without proper checks.

It should use either cpumask_setall? or use cpu_online_mask?

Your current patch rcu_cpu_beenfullyonline indicates that code around srcu_schedule_cbs_sdp handles hotplug already right?
in that case, just setting mask = cpu_online_mask would work?

Agreed. Which is why I have this commit queued:

f8d5aaaf90f8 ("srcu: Don't queue workqueue handlers to never-online CPUs")

This is currently slated for the upcoming merge window, but if you need it sooner, please let us know. Please see the end of this email for the full commit.

Thanx, Paul

Quoted 16 lines

Quoted 6 lines

/**
* queue_work_on - queue work on specific cpu
* @cpu: CPU number to execute work on
* @wq: workqueue to use
* @work: work to queue
*

Quoted 8 lines

* We queue the work to a specific CPU, the caller must ensure it
* can't go away. Callers that fail to ensure that the specified
* CPU cannot go away will execute on a randomly chosen CPU.
* But note well that callers specifying a CPU that never has been
* online will get a splat.
*
* Return: %false if @work was already on a queue, %true otherwise. */

Quoted 8 lines

In that case, making offline CPUs have a unbound workqueue is wrong. no?

It might encourage more users to abuse queue_work_on interface to send to offline CPUs without any checks and onus now falls onto workqueue to disaptch to unbound wqs.

So I think it is better to put the guardrails in SRCU instead of any change in workqueue.

------------------------------------------------------------------------

commit f8d5aaaf90f8294890802ce8dccbafd9850ac5f9

Quoted 8 lines

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 9 11:16:02 2026 -0700

srcu: Don't queue workqueue handlers to never-online CPUs While an srcu_struct structure is in the midst of switching from CPU-0 to all-CPUs state, it can attempt to invoke callbacks for CPUs that have never been online. Worse yet, it can attempt in invoke callbacks for CPUs that never will be online due to not being present in the

for CPUs that never will be online due to being present in the cpu_possible_mask?

Quoted 36 lines

cpu_possible_mask. This can cause hangs on s390, which is not set up to deal with workqueue handlers being scheduled on such CPUs. This commit therefore causes Tree SRCU to refrain from queueing workqueue handlers on CPUs that have not yet (and might never) come online. Because callbacks are not invoked on CPUs that have not been online, it is an error to invoke call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() on a CPU that is not yet fully online. However, it turns out to be less code to redirect the callbacks from too-early invocations of call_srcu() than to warn about such invocations. This commit therefore also redirects callbacks queued on not-yet-fully-online CPUs to the boot CPU.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Tested-by Vasily Gorbik <[email protected]>

Cc Tejun Heo <[email protected]>

kernel/rcu/srcutree.c +3 -5

index 0d01cd8c4b4a7b..7c2f7cc131f7ae 100644
@@ -897,11 +897,9 @@ static void srcu_schedule_cbs_snp(struct srcu_struct *ssp, struct srcu_node *snp
  {
  	int cpu;
-	for (cpu = snp->grplo; cpu <= snp->grphi; cpu++) {
-		if (!(mask & (1UL << (cpu - snp->grplo))))
-			continue;
-		srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, cpu), delay);
-	}
+	for (cpu = snp->grplo; cpu <= snp->grphi; cpu++)
+		if ((mask & (1UL << (cpu - snp->grplo))) && rcu_cpu_beenfullyonline(cpu))
+			srcu_schedule_cbs_sdp(per_cpu_ptr(ssp->sda, cpu), delay);
  }
  /*

@@ -1322,7 +1320,9 @@ static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp,
  	 */
  	idx = __srcu_read_lock_nmisafe(ssp);

kernel/rcu/srcutree.c static unsigned long srcu_gp_start_if_needed(struct srcu_struct *ssp, · line 1325

Quoted 5 lines

kernel/rcu/srcutree.c +3 -1

  	ss_state = smp_load_acquire(&ssp->srcu_sup->srcu_size_state);
-	if (ss_state < SRCU_SIZE_WAIT_CALL)
+	// If !rcu_cpu_beenfullyonline(), interrupts are still disabled,
+	// so no migration is possible in either direction from this CPU.
+	if (ss_state < SRCU_SIZE_WAIT_CALL || !rcu_cpu_beenfullyonline(raw_smp_processor_id()))

How can this happen? To get a CPU offline in raw_smp_processor_id() you need to run on the offline CPU.

Quoted 3 lines

kernel/rcu/srcutree.c +0 -0

  		sdp = per_cpu_ptr(ssp->sda, get_boot_cpu_id());
  	else
  		sdp = raw_cpu_ptr(ssp->sda);

Hi Paul.

On 4/29/26 11:31 PM, Paul E. McKenney wrote:

Quoted 6 lines

------------------------------------------------------------------------

commit f8d5aaaf90f8294890802ce8dccbafd9850ac5f9

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 9 11:16:02 2026 -0700

Quoted 5 lines

srcu: Don't queue workqueue handlers to never-online CPUs While an srcu_struct structure is in the midst of switching from CPU-0 to all-CPUs state, it can attempt to invoke callbacks for CPUs that have never been online. Worse yet, it can attempt in invoke callbacks for CPUs that never will be online due to not being present in the

for CPUs that never will be online due to being present in the cpu_possible_mask?

Quoted 15 lines

cpu_possible_mask. This can cause hangs on s390, which is not set up to deal with workqueue handlers being scheduled on such CPUs. This commit therefore causes Tree SRCU to refrain from queueing workqueue handlers on CPUs that have not yet (and might never) come online. Because callbacks are not invoked on CPUs that have not been online, it is an error to invoke call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() on a CPU that is not yet fully online. However, it turns out to be less code to redirect the callbacks from too-early invocations of call_srcu() than to warn about such invocations. This commit therefore also redirects callbacks queued on not-yet-fully-online CPUs to the boot CPU.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Tested-by Vasily Gorbik <[email protected]>

Cc Tejun Heo <[email protected]>

Quoted 3 lines · Shrikanth Hegde

Hi Paul.

On 4/29/26 11:31 PM, Paul E. McKenney wrote:

[ . . . ]

Sorry, missed one...

Quoted 6 lines

Quoted 6 lines

------------------------------------------------------------------------

commit f8d5aaaf90f8294890802ce8dccbafd9850ac5f9

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 9 11:16:02 2026 -0700

Quoted 7 lines

Quoted 5 lines

srcu: Don't queue workqueue handlers to never-online CPUs While an srcu_struct structure is in the midst of switching from CPU-0 to all-CPUs state, it can attempt to invoke callbacks for CPUs that have never been online. Worse yet, it can attempt in invoke callbacks for CPUs that never will be online due to not being present in the

for CPUs that never will be online due to being present in the cpu_possible_mask?

Exactly.

Just because a CPU is in cpu_possible_mask doesn't mean that it will ever actually come online. For example, for single-threaded performance reasons, a given system might choose to bring online only one CPU from each hypertheaded core. In that case, the other CPU in each hyperthreaded core could be in the cpu_possible_mask, but would never come online.

Thanx, Paul

Quoted 7 lines

Quoted 7 lines

cpu_possible_mask. This can cause hangs on s390, which is not set up to deal with workqueue handlers being scheduled on such CPUs. This commit therefore causes Tree SRCU to refrain from queueing workqueue handlers on CPUs that have not yet (and might never) come online. Because callbacks are not invoked on CPUs that have not been online, it is an error to invoke call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() on a CPU that is not yet fully online.

Quoted 8 lines

Quoted 8 lines

However, it turns out to be less code to redirect the callbacks from too-early invocations of call_srcu() than to warn about such invocations. This commit therefore also redirects callbacks queued on not-yet-fully-online CPUs to the boot CPU.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Tested-by Vasily Gorbik <[email protected]>

Cc Tejun Heo <[email protected]>

Hi Paul.

Quoted 25 lines · Paul E. McKenney

Quoted 3 lines · Shrikanth Hegde

Hi Paul.

On 4/29/26 11:31 PM, Paul E. McKenney wrote:

[ . . . ]

Sorry, missed one...

Quoted 6 lines

Quoted 6 lines

------------------------------------------------------------------------

commit f8d5aaaf90f8294890802ce8dccbafd9850ac5f9

Author: Paul E. McKenney <[email protected]>
Date:   Thu Apr 9 11:16:02 2026 -0700

Quoted 7 lines

Quoted 5 lines

srcu: Don't queue workqueue handlers to never-online CPUs While an srcu_struct structure is in the midst of switching from CPU-0 to all-CPUs state, it can attempt to invoke callbacks for CPUs that have never been online. Worse yet, it can attempt in invoke callbacks for CPUs that never will be online due to not being present in the

for CPUs that never will be online due to being present in the cpu_possible_mask?

Exactly.

Quoted 5 lines

Just because a CPU is in cpu_possible_mask doesn't mean that it will ever actually come online. For example, for single-threaded performance reasons, a given system might choose to bring online only one CPU from each hypertheaded core. In that case, the other CPU in each hyperthreaded core could be in the cpu_possible_mask, but would never come online.

Nit: I was suggesting *not* is probably not needed in that changelog. I agree with explanation.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

cpu_possible_mask. This can cause hangs on s390, which is not set up to deal with workqueue handlers being scheduled on such CPUs. This commit therefore causes Tree SRCU to refrain from queueing workqueue handlers on CPUs that have not yet (and might never) come online. Because callbacks are not invoked on CPUs that have not been online, it is an error to invoke call_srcu(), synchronize_srcu(), or synchronize_srcu_expedited() on a CPU that is not yet fully online.

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

However, it turns out to be less code to redirect the callbacks from too-early invocations of call_srcu() than to warn about such invocations. This commit therefore also redirects callbacks queued on not-yet-fully-online CPUs to the boot CPU.

Reported-by Vasily Gorbik <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Tested-by Vasily Gorbik <[email protected]>

Cc Tejun Heo <[email protected]>

Alright. With those two explanations, this LGTM.

Reviewed-by Shrikanth Hegde <[email protected]>